Skip to content. | Skip to navigation

You are here: Home content generated bystar PLPC 180051 current accessPage

Multi Account Resident Mail Exchanger (MARME) And Environment (MARMEE)

Multi Account Resident Mail Exchanger (MARME) And Environment (MARMEE)

A Permanent Libre Published Content

Multi Account Resident Mail Exchanger Environment (MARMEE)

Based On X822-MSP (Mail Submission Pipeline)

--- A BISOS Feature-Area And A ByStar Core Capability

Document Number: PLPC-180051   [ .bib ]
Version: 0.4
Dated: May 7, 2018
Group: facilities
Primary URL:
Federated Publications: ByTopic -- ByContent
AccessPage Revision: This AccessPage was produced on May 07, 2018 at 23:13 PDT (-0700)
Author(s): Neda Communications, Inc.


  • PDF: -- 816K -- Provides the document in Portable Document Format.
  • HTML: -- 240K -- Displays the document as a web page.



Multi Account Resident Mail Exchanger (MARME) is a general purpose autonomous software package that implements mail sending and mail processing facilities. MARME can be augmented by a resident MTA and MUAs such as the Emacs Mail User Environment (Blee-Msg) producing Multi Account Resident Mail Exchanger Environment (MARMEE).


Multi Account Resident Mail Exchanger (MARME) And Environment (MARMEE)

Multi Account Resident Mail Exchanger Environment (MARMEE)
Based On X822-MSP (Mail Submission Pipeline)
— A BISOS Feature-Area And A ByStar Core Capability

Document #PLPC-180051
Version 0.4
May 7, 2018
This Document is Available on-line at:

Neda Communications, Inc.


Chapter 1  About MARMEE

1.1  About Multi Account Resident Mail Exchanger Environment (MARMEE)

Multi Account Resident Mail Exchanger (MARME) is a collection of Linux facilities that provide for rich sending and receiving of email.

MARME is then augmented by a resident MTA (Mail Transfer Agent) and MUAs (Mail User Agents) such as the Emacs Mail User Environment (Blee-Msg) producing Multi Account Resident Mail Exchanger Environment (MARMEE).

Uses of MARMEE include:

  1. Reliable Application Integrated Tracked Mail Sending For Confirmed Communications
  2. Email Marketing (Similar to Constant Contact capabilities)
  3. Device resident complete mail sending, mail receiving and mail processing environments

MARME tracks delivery of email and allows for detection and processing of delivery reports, receipt notifications and bounced messages.

MARME integrates the following major software packages:

offlineimap (Python Commands Package):
Used for retrieving and synchronizing email from remote servers
notmuch (Python Commands Package):
Used for searching and reading messages
flufl.bounce (Python Library Package):
Identifies bounced messages and isolates original message within bounced messages
email (Python Library Package):
A library for managing email messages, including MIME and other RFC 2822-based [1] message documents.
smtplib (Python Library Package):
Defines an SMTP client session object that can be used to send mail to any Internet machine with an SMTP or ESMTP listener daemon.
msgOut from bxMsg (ByStar Python Library Package):
Builds on email and smtplib for outgoing email.
msgIn from bxMsg (ByStar Python Library Package):
Builds on email for incoming email.
msgLib from bxMsg (ByStar Python Library Package):
Builds on email for common aspects of incoming and outgoing email.

Based on the orchestration of these major software packages MARME manages sending and tracking of sent messages.

1.2  Outline Of This Document

The outline of this document is illustrated in Figure 1.1.

Figure 1.1: MARME Layerings

Part I
MARMEE Software Installation, Configuration, Control And Operation

Chapter 2  MARMEE Software Installation And Configuration

2.1  Installing This Software Package In BISOS Or Independently

There are multiple ways to obtain and installing and configuring this software package.

This software package is part of BISOS. See BISOS for details.

You can install this software package in /bisos based on BISOS policies or you can install it as an independent standalone software package.

This documentation primarily focuses on installation and configuration in /bisos based on BISOS policies.

2.2  BISOS Bases Installation

The root of installation of BISOS is: $bisosBase/bisos

Often, $bisosBase/bisos is /bisos. When we refer to /bisos we are referring to $bisosBase/bisos.

If /bisos has not been installed, we need to prepare the /bisos bases.

2.2.1  Basic Preparation

There are two ways of setting up the BISOS bases.

The setup with Python 2.7 default installation is simpler.

It is also possible to setup BISOS bases without intruding on the existing default python environment.

BISOS Bases Setup – Without Default Python 2.7 – Non-Intrusive To Default Python Environment

If Python 2.7 is not installed on your system, install it. Python 2.7 need not become the default python.

When Python 2.7 is the default installation, this mode of installation does not impact the default python environment. Everything runs under the /bisos/venv/py2-bisos-3 virtual environment.

pip install virtualenv
sudo mkdir -p /bisos/venv/py2-bisos-3
sudo chown -R aUser:aGroup /bisos
virtualenv --no-site-packages --python=python2.7 /bisos/venv/py2-bisos-3
source /bisos/venv/py2-bisos-3/bin/activate
pip install --no-cache-dir --upgrade bisos.bx-bases
bx-bases -v 20 --baseDir=/bisos  -i pbdUpdate all

BISOS Bases Setup – With Python 2.7 Default

pip install virtualenv
sudo mkdir -p /bisos
sudo chown -R aUser:aGroup /bisos
pip install --no-cache-dir --upgrade bisos.bx-bases
bx-bases -v 20 --baseDir=/bisos  -i pbdUpdate all

/bisos/venv/py2-bisos-3 virtenv is then created.

2.2.2  BISOS CORE Pip Installs In Virtenv

With /bisos base in place, we can now add BISOS Core.

source /bisos/venv/py2-bisos-3/bin/activate

Then add a BISOS feature-area (BISOS package)

For example:

pip install --no-cache-dir --upgrade bisos.core   # NOTYET

2.3  BISOS Bases Directory Structure Overview

BISOS base directories are created with bx-bases.

In with

def pbdDict_bisosRoot(baseDir,):

the directory structure is created.

Assuming /bisos as root, this is a brief description of the directory hierarchy.

Current distributions are accessible from this base. A distribution could have been obtained from pip, or through AnonVC, or through AuthVC.
The distribution obtained through pip. Structured through symlinks to /bisos/venev
Links to the current bisos version
A base for all virtual environments.
Obtained with:
virtualenv --no-site-packages --python=python2 /bisos/venv/py2-bisos-3
Obtained with:
virtualenv --no-site-packages --python=python2 /bisos/venv/py3-bisos-3
A base for Git/CVS/etc authenticated obtained dists.
A base for Git/CVS/etc anonymous obtained dists.
Base for symlinks to current selected (pip/vcAuth/vcAnon) bisos
Base for symlinks to current selected (pip/vcAuth/vcAnon) bsip
Base for symlinks to current selected (pip/vcAuth/vcAnon) blee

2.4  Optional Addition Of BISOS Feature-Areas

With /bisos base in place, you can now add your desired BISOS packages.

source /bisos/venv/py2-bisos-3/bin/activate

Then add a BISOS feature-area (BISOS package)

For example:

pip install --no-cache-dir --upgrade unisos.marme
pip install --no-cache-dir --upgrade bisos-gossonot

2.5  MARMEE Installation And Configuration As A BISOS Software Package

With /bisos bases in place, we can now install unisos.marme and configure it for usage in the /bisos environment.

2.5.1  MARMEE Software Package Installations In BISOS

We will next install unisos.marme in /bisos/venv/py2-bisos-3:

source /bisos/venv/py2-bisos-3/bin/activate
pip install --no-cache-dir --upgrade unisos.marme

2.5.2  MARMEE Software Package Configuration In BISOS

unisos.marme software package receives its run base configuration parameters from a series of FileParameters (fp) in:


Necessary tools are provided to configure and inspect these configurations as Parameters.


provides you a menu of commands that you can run to configure the unisos.marme package.

You can inspect the current values by running: -v 20
  -i pkgInfoParsGet 

MARMEE Control Base Specification

In the BISOS environment most configuration parameters can be determined based on defaults, but the MARMEE Control Base needs to be explicitly specified.

Assuming that MARME control files reside in $HOME/marmeControlBase

Default configuration within BISOS can be set with: -v 20
 --icmsPkgName="" --icmsPkgControlBaseDir="${HOME}/marmeControlBase"
  -i pkgInfoParsDefaultsSet bisosPolicy

2.5.3  MARME Software Preparations

In addition to the MARME Software Package itself, MARME uses various common packages and libraries. These are obtained and managed by running: -v 20 -i binsPreps

2.6  MARMEE Installation And Configuration As An Idependent Software Package

You can also install MARMEE independent of BISOS.

Such an independent and configuration involves creating the necessary control bases similar to how they are done within BISOS.

Chapter 3  MARMEE Software Control And Operation

3.1  MARME Control FileParameters –

MARME is controlled through a set of FileParameters residing at the path specified by the –icmsPkgControlBaseDir option of

These FileParameters are usually configured and inspected by


provides a menu for setting the parameters.

These options and parameters are described below.

Prior to the execution of MARAME facilities, you should verify that the needed parameters are properly set.

3.1.1  MARME Identifiers and Control Parameters

MARME operates based on a set of parameters and identifiers which we enumerate below.

  /AbstractionTerminology/:: mailAcctDefault, inMailAcct, outMailAcct
  mailAcctName             :: Name for a mailAcct which can be inMailAcct or outMailAcct or both
  mailAcctCur              :: Currently Slected mailAcct (drives inMailAcct and outMailAcct)

  outMailAcct              :: Name of outgoing mail account (smtpServer)
  outMailAcctControlerPars :: Control FPs for outgoing mail account owner (firstName, lastName)
  outMailAcctAccessPars    :: Control FPs for outgoing mail account (smtpServer)

  inMailAcct               :: Name of incoming mail account (imapServer)

  inMailAcctAccessPars     :: Control FPs for incoming mail account (imapServer)
  inMailAcctControlerPars  :: Control FPs for incoming mail account owner (firstName, lastName)
  inMailAcctRetrievePars   :: Control FPs for incoming mail account -- What folders to bring and where to put them
  inMailAcctMboxesPath     :: Base directory of all inMailAcct Mailboxes
  inMailAcctInbox          :: (maildir) Base directory of inMailAcct Inbox
  inMailAcctMboxCur        :: (maildir) Base directory of currently selected inMailAcct Mbox

3.1.2  Control And Config Structures And Usage

/File Bases/
mailAcctsBaseDir         :: ../
controlBaseDir           :: ../control/  -- common,inMail/mailAcctName,outMail/mailAcctName
configBaseDir            :: ../conf/     -- ../conf/mailAcctName/_configName
varBaseDir               :: ../var       -- ../var/inMail/mailAcctName/maildir, ../var/outMail/mailAcctName/{log,msgs}
tmpBaseDir               :: ../tmp/.

inMailAcctAccessBase     :: join(controlBaseDir, "inMail", inMailAcct)
inMailAcctMboxesBase     :: join(varBaseDir, "inMail", inMailAcct, "maildir")
inMailAcctInbox          :: join(inMailAcctMboxesBase, "Inbox")

The default profile is chosen by:

echo "mailDom" > ${marmeBase}/control/common/defaultMailDom/value

3.2  MARME Software Package Operation

All MARME scripts, programs and tools are Interactive Command Modules (ICM). See for details.

All MARME scripts, programs and tools are written in the COMEEGA (Collaborative Org-Mode Enhanced Emacs Generalized Authorship) style. Emacs and org-mode are needed for these enhancements.

The following control and informational tools are in the marmeBase/bin.

3.2.1  Control and Informational Tools – Control Base            == Install ICMs-Pkg dependencies          == Configure and Manage Mail Svcs            == offlineimap based on inMailBase           == notmuch on inMail-Maildirs             == Act on DSNs               == Based on msgOut

3.3  MARME Software Interfaces and Usage

3.3.1  Mail Submission/Injection Rules/Logic

Four sets of improvements are shown in this figure:

  1. A dedicated mailbox “” – separate from admin@.
  2. Improved Mail Injection Rules – top right oval.
  3. “DSN /NDR (bounce) processor” – top left oval.
  4. A Message Tracking database

Two sets of interfaces are also shown in this figure:

  1. A interface
  2. A interface

We expand on these below.

Part II
X822-MSP (Mail Submission Pipeline)

Chapter 4  X822-MSP (Mail Submission Pipeline)

4.1  X822-MSP (Mail Submit Pipeline)

BX-Tags vs X-B-Tags is that BX-Tags are consumed and stripped at MailSending phase but X-B-Tags are kept and may be visible to the recipient and when bounced.

The X-B- header lines are public and should not be stripped. The B-X- header lines are private and will be stripped.

4.1.1  X822-MSP: Mail Gui To Mail Submit Client Software Pipeline

 msgOut is a library (set of facilities) that provide for
  composition and submission of email messages based on the
  *Mail Gui To Mail Submit Client Software Pipeline*
  During mail-composition, a number of mail-headers are
  added to the email header.
  When the email is to be sent, all the necessary information
  for the mail submission client is within the email headers.
  Standard capabilities of X822 Mail Submit Pipeline (X822-MSP) are:
    - Envelope-Addr specification
    - Deleivery-Status-Notifications Request  (bounce addresses and delivery reports)
    - Disposition-Notifications  (read-receipts)
    - Flexible Parameterized Message Submission (host, ssl, user, passwd)
  Sending is the act of delivering the message to another
  process for the purpose of transfer.
  Sending can be one of:
  Injection -- using the command line and pipes
  Submission -- using a protocol (smtp, etc.)
 :CompositionModel: (Mail User Agent)
 The msg itself is used as a container to gether and carry
 all parameters and all requests for the message submission.
 The following local header fields are recognized:
 BX-Sending-Method:          # inject, submit
 BX-Sending-Run-Control:   # dryrun, debug
BX-MTA-Injection -- Obsoleted
 BX-MTA-Injection-Plugins:   # for composite injection profile
 BX-MTA-Injection-Method:    # inject, submit
 BX-MTA-Injection-Control:   # dryrun, debug
 BX-MTA-Rem-Protocol:        # smtp
 BX-MTA-Rem-LinkConfidentiality:  ssl/tls
 BX-MTA-Submission-Pre-Plugins:    # executed before send
 BX-MTA-Submission-Post-Plugins:   # executed after send for error reporting
SubmissionModel: (Mail Transfer Agent)
BX-MTA-Submission-Pre-Plugins are executued in order specified.
All the BX- headers are recognized and converted to params
Where appropriate BX- headers are converted to standard headers.
Some BX- headers are stripped
Complete SMTP Submit Protocol based on the email.smtp python library is executed.
BX-MTA-Submission-Post-Plugins are executued in order specified.

Part III
X822 Msg Library

Chapter 5  X822 Msg Library

5.1 Library

5.1.1 – Interface And Implementation Interface

A complete interface for email-sending that supports delivery-notification-requests, non-delivery-notification-requests, and receipt-notification-requests has been defined and implemented.

Primary functions of that interface are listed below.

def dispositionNotificationRequetsForTo(
    """Request Receipt-Nofications for each of recipientsList. Notifications are to be sent to notifyTo address."""
def deliveryNotificationRequetsForTo(
    """Request Delivery-Reports for each of recipientsList. Notifications are to be sent to notifyTo address."""

def nonDeliveryNotificationRequetsForTo(
    """Request Non-Delivery-Reports for each of recipientsList. Notifications are to be sent to notifyTo address."""

def nonDeliveryNotificationActions(
    """Based on action and address, process the Non-Delivery-Notification and notify each of coRecipientsList."""

def envelopeAddrSet(
    """Set the msg's envelope address to mailBoxAddr. This will be used for delivery-reports and non-delivery-reports."""

def crossRefInfo(
    """Set the crossRefInfo. E.g., peepid."""

def injectionParams(
        injectionMethod,         # submission, qmail-inject
        mtaRemProtocol,          # smtp
        mtaRemHost,              # Remote Host To Submit to (could be localhost)
    """Set necessary  injectionParams, to be used by injectBasedOnHeadersInfo."""

def injectBasedOnHeadersInfo(
    """Submit or Inject msg using information contained in the header. Implementation implements delivery-notification-requests, non-delivery-notification-requests, and receipt-notification-requests.

In the header fields of out going messages we can include:

Notice-Requested-Upon-Delivery-To: <enroleeAddress>

Additionally with smtplib at the time of:

SMTP.sendmail(from_addr, to_addrs, msg[, mail_options, rcpt_options])

The rcpt_options can be used.

Based on the above, most of what can reasonably be done to receive a positive delivery report has been accomplished. Usage

An example of usage of is included in

sendmimeemail can be updated to use

Part IV
MARME – Overview

Chapter 6  MARME Overview

6.1  MARME Model, Abstractions And Terminology

The MARME model fully separates incoming mail retrieval (inMail) from mail sending functionality (outMail). After an email is sent, its delivery and final receipt is tracked through automated retrieval of Delivery Status Notifications (DSN).

6.1.1  Tracked Mail Sending Applications Framework Overview

The general framework for sending and tracking is shown in Figure 6.1.

Figure 6.1: Multi Account Resident Mail Exchanger (MARME) Architecture Overview

The organization of components of MARME is described below.

Mail Sending Applications

The mail sending application uses X822-MSP (Mail Submit Pipeline) tags for sending of emails and then delivers it to msgOut.

Mail Submission/Injection Interface (msgOut from bxMsg)

Mail Submission/Injection Interface is of the bxMsg library.

msgOut accepts emails that are augmented by X822-MSP (Mail Submit Pipeline) tags and submits/injects them based on that information.

Mail Sending Agent (SMTP Client)

msgOut then processes X822-MSP tags and uses smtplib to submit the message.

smtplib includes a complete SMTP implementation. Shown in Figure 6.1 labeled as “Mail Sending Agent (SMTP Client)”.

SMTP Submit Server

The address of SMTP Submit Server and the needed credentials are specified in X822-MSP tags which then point to the selected SMTP Submit Server.

A Dedicated DSN (Delivery Status Notification) Address and Mailbox

At submission time, Delivery Status Notification requests are directed to be sent to a dedicated DSN address and mailbox such as:

Bounces Mailbox

Bounce messages (both due To: and Cc: bad addresses) will then end up at mailbox of

This is shown on the left side of Figure 6.1.

Chapter 7  MARME - Delivery Tracking

7.1  Message Transition Tracking – Interface And Implementation

As information about delivery/non-delivery of outgoing emails is observed, they will be logged in a database.

The state transition table below captures all recognized states and transitions.

Figure 7.1: Mail Tracking States

7.1.1  Message Transition Tracking Python Interface

Based on the above state definitions, a simple interface – shown below – can be used to capture tranistions.

def msgStateTransitionLog( msg, currentState, nextState, transitionInfo=None, ): """Log a significant state change with regard to delivery of a message."""

7.1.2  Message Transition Tracking Python Implementation – Application Database Integration

Based on the above interface, the mail tracking state transitions need to be logged in appropraite databases.

Chapter 8  MARME DSN Processor

8.0.3  DSN /NDR (bounce) Processor

Based on the above all delivery related messages will be sent to the dedicated

These include:

  • Failed Delivery (NDR) – Non-Delivery Reports (bounces)
  • Successful Delivery – In response to DSN Requests
  • Delayed Delivery due to temporary failures – soft failures
  • Receipt Notifications – Read receipts

These can then be auto processed.

The contours of how this is being done is outlined below:

  • A daemon will imap poll account/mbox for new messages.
  • For each positive delivery, a log is created
  • For each hard failure (bounce), the failed address is recognized and a copy of the failure is sent to other recipients of the same message.
  • For each soft failure (temporary failure) a log is made.

8.0.4  Automated NDR Notification Of Co-Recipients

Consider the situation where John sends and email to Alice and to (or CCs) Bob. Alice’s email address results in a non-delivery and John receives a non-delivery-report (NDR). But, Bob has no knowledge of this NDR and may behave based on the assumption that Alice has received John’s email.

MARME provides for optional NDR notification of co-recipients. Upon receipt of the NDR by John, Bob can be automatically notified that Alice did not receive John’s email.

8.0.5  MARME DSN Overview

The general framework for sending and tracking is shown in Figure 8.1.

Figure 8.1: MARME Delivery Status Notification (DSN) Processor

Chapter 9  Mail Sending Application Examples

9.1  Confirmed Notifications Email Delivery Applications

A common class of applications that can be built with MARME are “Confirmed Notifications Email Delivery Applications”.

An example could be email delivery of offer letters in a human resources applications.

9.2  Email Marketting Applications

A common class of applications that can be built with MARME are “Email Marketting Applications”.

MARME is capable of providing most of the capabilities that the likes of Constant Contact offer.

Part V
MARME User Environment – MARMEE

Chapter 10  MARME User Environment

10.1  Multi Account Resident Mail Exchanger Environment (MARMEE)

10.1.1  Device software integration

We use an architecture based on the concept of a Device-Resident End-MTA middleware module, acting as intermediary between the protocol software and the MUA.

Figure ?? shows the software architecture for integration of EMSD-UA with qmail to create a Device-Resident End-MTA. On its external interface (shown in grey and yellow at the bottom of the figure), the Device-Resident End-MTA interacts with the Internet at large using SMTP, and IMAP. On its internal interface (local loop-back interface; address the Device-Resident End-MTA interacts with the MUA based on SMTP and IMAP. Thus the MUA need have no awareness of EMSD at all. This architecture is quite general and can be used on almost all platforms. In this model, the MUA is always configured for the interface for the SMTP gateway, and the IMAP server. The Device-Resident End-MTA is then configured with the real external server information.

offlineimap is used to optionally synchronize the device’s mailstore/Maildir (shown in grey) so that the user’s inbox is locally available, even when there is no network connectivity.

Figure 10.1: General Mail User Environment (Marmee)

Though this architecture is based on qmail, the resulting Device-Resident End-MTA package is quite general, and can be installed in all Linux PDA platforms, and very likely other platforms too.

Note that because all software shown in Figure ?? is free/Libre software, the Device-Resident End-MTA can be made available on any Linux-based device without any restrictions.

Part VI

Chapter 11  ByStar MARMEE

11.1  ByStar MARMEE

In what was presented above the remote smtp server and the remote imap server were

Figure 11.1: ByStar Mail User Environment (BxMarmee)


P. Resnick. Internet Message Format. RFC 2822 (Proposed Standard), April 2001. Obsoleted by RFC 5322, updated by RFCs 5335, 5336.

Document Actions
Libre/Halaal Internet Services Provided At LibreCenter By Neda

Member of By* Federation Of Autonomous Libre Services

This web site has been created based exclusively on the use of Halaal Software and Halaal Internet Application Services. It is part of the By* Federation of Autonomous Libre Services which in turn are part of the Halaal/Libre By* Digitial Ecosystem which incorporate the following software components: