Oct 22

Python Tips for SAP BODS

Some months ago I had to implement a user defined transform (UDT) for SAP Business Object Data Services in Python. We integrated social media services into a SAP based platform and there are a few things I stumbled upon while trying to implement it.

This post is meant as an extension to the excellent post by Jake Bouma on Better Python Development for BODS: How and Why.

Read the Manual

Read chapter 11 of the official manual of SAP for the topic which is the best and only source of information you have. There is also an official blog plog giving a high level overview.

Develop Locally

The first step to being more productive is testing and developing the UDT on your local machine and only deploy it when it is ready. The BODS Python editor is unusable for development because you need a fast feedback loop when developing your program. Look at the BODS editor as deployment method for your Python code.

In order to execute your UDT on the local machine and on the SAP platform you need to design it like regular Python programs and mock out the SAP interface for testing.

Design UDTs as regular Python programs

The official SAP documentation is not a good example for idiomatic Python code.

Structure your script into functions and an entrypoint - just like you would do with a local command line tool. Keep the SAP integration separate to keep your code testable.

def insert_newest_tweets():
   print 'Fetching newest tweets'
   tweets = fetch_newest_tweets('@lukmartinelli')
   for tweet in tweets:
       create_record(tweet)
       print 'Successfully added {} to collection'.format(tweet.id)
   print 'Finished collecting {} tweets'.format(len(tweets))


if __name__ == '__main__':
    insert_newest_tweets()

The __name__ == '__main__' clause will check if the code is executed as standalone program or imported as library.

Check the interpreter

You can test in your program whether you are running inside SAP or an other environment by checking the interpreter that executes your script.

import sys

RUNS_IN_SAP = sys.executable.endswith(u'al_engine.exe')

This allows you to import the mocked SAP interface in case you are not running on the SAP platform.

if not RUNS_IN_SAP:
    from sap import Collection, DataManager

Another example is configuring a corporate proxy when code is running in production.

if RUNS_IN_SAP:
    PROXY = u'proxy.corp.local:8080'

Emulate substitution parameter

If you are using substitution parameters in production you can emulate those with environment variables. Substitution parameters are replaced at runtime by the SAP Python interpreter.

This function checks whether the value has been replaced and if not returns default value.

def load_substitution_param(substitution, default_value):
    if substitution.startswith('[$$'):
        return default_value
    else:
        return substitution

Using it together with local environment variables.

POST_LIMIT = int(load_substitution_param(r'[$$FACEBOOK_POST_LIMIT]',
                                         os.environ.get('POST_LIMIT', 200)))

Mock out the SAP interface

To be able to unit test your code you need to mock out the SAP interface. The easiest way is to create a new file sap.py and import it only when running locally (like described above).

if not RUNS_IN_SAP:
    from sap import Collection, DataManager

I’ve written some code that mimics the BODS Collection and Record functionality. The classes are exported as global singletons so you can use them in the same fashion as the normal API.

You can use the mocked interface to write unit tests or read from stdin for testing your code locally.

"""
This module enables local testing of python code
used for User Defined Transforms in SAP BODS
"""

class FIDataCollection:
    records = []

    def AddRecord(self, record):
        """
        Add a new record(returned by DataManager.NewDataRecord()) to the
        collection.
        For every NewDataRecord(), you can call AddRecord() only once.
        After you call AddRecord(), do not call DeleteDataRecord().
        """
        self.records.append(record)

    def DeleteRecord(self, record):
        """Removes the specified record from collection."""
        self.records.remove(record)

    def GetRecord(self, record, index):
        """Get a record at a given index from the collection."""
        if index < 1:
            raise ValueError("Indizes in SAP BODS start at 1 and not 0!")
        for key, value in self.records[index-1].values.iteritems():
            record.SetField(key, value)

    def Size(self):
        """
        Counts the number of records in the collection.
        Returns number of records in a collection.
        """
        return len(self.records)

    def Truncate(self):
        """Removes all the records from collection."""
        self.records = []


class FIDataManager:
    def NewDataRecord(self, ownership=1):
        """
        Creates a new record object. Do not use this method in a loop,
        otherwise the Python expression may experience a
        memory leak. Depending on the expression, you'll probably want to
        place this method at the beginning of the expression.
        Returns a new object of type FlDataRecord.
        """
        return FIDataRecord()

    def DeleteDataRecord(self, record):
        """
        Deletes the memory allocated to the record object.
        Do not call DeleteDataRecord() after calling AddRecord().
        """
        del record


class FIProperties:
    values = {}

    def GetProperty(self, property_name):
        """Returns the value of given property."""
        if not isinstance(property_name, unicode):
            raise ValueError("You must use unicode for property name")
        return self.values[property_name]


class FIDataRecord:
    values = {}

    def SetField(self, field_name, value):
        """Stores a value in the specified field."""
        if not isinstance(field_name, unicode):
            raise ValueError("You must use unicode for field name")
        if not isinstance(value, unicode):
            raise ValueError("You must use unicode for setting the value")
        self.values[field_name] = value

    def GetField(self, field_name):
        """Get a field from record."""
        if not isinstance(field_name, unicode):
            raise ValueError("You must use unicode for field name")
        return self.values[field_name]


Collection = FIDataCollection()
DataManager = FIDataManager()
Properties = FIProperties()

Other Tricks

Timestamp format for SAP

Format a Python time object in a SAP compatible format.

def sap_timestamp(timestamp):
    return time.strftime('%Y.%m.%d %H:%M:%S', timestamp)

Keep Business logic and SAP integration separate

Create a class in Python or use a namedtuple and write the mapping to the data records separately.

class Tweet(object):
    def __init__(self, screen_name, created_at, id, text, favourites,
                 retweets, followers, friends):
        self.screen_name = screen_name
        self.created_at = created_at
        self.id = id
        self.text = text
        self.favourites = favourites
        self.retweets = retweets
        self.followers = followers
        self.friends = friends

And now create a help function to create the record.

def create_record(tweet):
    rec = DataManager.NewDataRecord(1)

    rec.SetField(u'SCREEN_NAME', unicode(tweet.screen_name))
    rec.SetField(u'TIME_STAMP', unicode(sap_timestamp(tweet.created_at)))
    rec.SetField(u'TWEET_ID', unicode(tweet.id))
    rec.SetField(u'TWEET_TEXT', unicode(tweet.text))
    rec.SetField(u'FAV_COUNT', unicode(tweet.favourites))
    rec.SetField(u'RETWEETS', unicode(tweet.retweets))
    rec.SetField(u'FOLLOWERS_COUNT', unicode(tweet.followers))
    rec.SetField(u'FRIENDS_COUNT', unicode(tweet.friends))

    Collection.AddRecord(rec)

    # We need to remove the rec reference because it is an external object
    del rec

Conclusion

I hope you can use these tricks when creating UDTs with Python. Remember that writing a separate Python program to integrate other services into SAP is probably the better way but sometimes you are restricted by the environment you work in.