Manuel on Software - Reloaded http://mvaled.github.io/blog/html/ en-us Mon, 18 Nov 2019 00:00:00 -0500 http://mvaled.github.io/blog/html/2019/11/18/performance-of-python-list-comprehensions.html http://mvaled.github.io/blog/html/2019/11/18/performance-of-python-list-comprehensions.html <![CDATA[Why does creating a list comprehension is faster?]]> Why does creating a list comprehension is faster?

Yesterday, I was removing a list-comprehension as an argument to sum(). This is the diff:

- sum([p.price for p in self.products])
+ sum(p.price for p in self.products)

To show the original programmer my line of though I performed a little experiment with the following message and code:

List comprehensions as an argument to reduce-like functions are usually less efficient than using the generation expression itself. The reason is that Python creates the list just to discard it afterwards:

>>> def fib(n):
...     a, b = 1, 1
...     while a < n:
...         yield a
...         a, b = b, a + b
...

>>> %timeit sum([x for x in fib(100)])
2.61 µs ± 33 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

>>> %timeit sum(x for x in fib(100))
3.13 µs ± 18.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

I was surprised to see that the list comprehension was a little bit faster than the generator expression. So it seems that for short-enough lists, the implementation of sum() is quite fast.

I had to change the implementation of fib to control the amount of items to show my point:

>>> def fib(n):
...     a, b = 1, 1
...     for _ in range(n):
...         yield a
...         a, b = b, a + b

An still with 100 items, passing a list is faster:

>>> %timeit sum(fib(100))
14.2 µs ± 212 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

>>> %timeit sum(list(fib(100)))
16.4 µs ± 247 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

>>> %timeit sum([x for x in fib(100)])
18 µs ± 160 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Of course the differences are too small to draw final conclusions. And, of course, as the list grows larger it becomes slower:

>>> %timeit sum([x for x in fib(10**5)])
497 ms ± 84.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

>>> %timeit sum(x for x in fib(10**5))
329 ms ± 17.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

So I think I still prefer the version using the generation expression just to cover my self from the case where there are too many items to be held in memory.

(I didn’t push the commit, though.)

]]> Mon, 18 Nov 2019 00:00:00 -0500 http://mvaled.github.io/blog/html/2018/08/01/composing-iterator-returning-functions.html http://mvaled.github.io/blog/html/2018/08/01/composing-iterator-returning-functions.html <![CDATA[Composing iterator-returning functions]]> Composing iterator-returning functions

A few days ago I was reviewing a piece of Python code. I was looking for a bug, but in the process I found a very interesting function.

The system allows the user to “extend” the structure of Products by providing more attributes which can be used later on when creating Sale (or Purchase) Orders. The Product object is like a class defining the structure, and the items in Orders are the instances of such classes.

When trying to export a full catalog, each attribute implies a new row in the spreadsheet file. To avoid too much coupling, this process was modeled by a kind of seeded generation of every possible row. The algorithm started with a seed instance of a product without any attribute, and then it generated every possible attribute-complete instance by composing several functions that took a instance and returned a iterator of instances. Each function deals with a specific type of attribute, and simply copies those attributes in the instances being generated.

The function doing the generation of all possible instance was more or less like this:

def iter_product(initial, *funcs):
   if not funcs:
      yield initial
   else:
      f, *fs = funcs
      for x in f(initial):
         yield from iter_product(x, *fs)

That definition was OK, but I wondered if I could build just the composition of several functions returning iterators, so that I can reuse it with several initial objects.

A little incursion in Haskell

In order to test my Haskell, I did first a Haskell version. I started by trying to create a composition operator much like the (.) operator, which has type:

(.) :: (b -> c) -> (a -> b) -> a -> c

The type of our composition of iterator-returning functions would be:

infixr 9 >>.
(>>.) :: (b -> [c]) -> (a -> [b]) -> a -> [c]

The choice of (>>.) as the operator will become (I hope) evident. The most straightforward implementation and easy to understand is using the list-comprehension syntax:

g >>. f = \x -> [z| y <- f x, z <- g y]

Can we generalize this? Yes! The list is an instance of a Monad, defined as:

instance Monad [a] where
    return x = [x]
    m >>= f  = concat (map f m)

And list comprehensions can be easily rewritten using the do notation:

(>>.) :: Monad m => (b -> m c) -> (a -> m b) -> a -> m c
g >>. f = \x -> do{y <- f x; z <- g y; return z}

The monadic >>= operator is usually called the bind. It’s type is

Monad m => m a -> (a -> m b) -> m b

So, I think there’s a compact way to write our >>. operator. And, now you may start to see why I chose >>..

The do notation is just syntax-sugar over using >>= (or its brother >>). The rules are given here. So let’s transform our implementation. We start we our current definition:

\x -> do {y <- f x; z <- g y; return z}

And rewrite the do two times until there are no more:

\x -> let s1 y = do {z <- g y; return z} in f x >>= s1

\x -> let s1 y = (let s2 z = return z in g y >>= s2) in f x >>= s1

Now, we can recall the eta-conversion rule and see that s2 = return, so:

\x -> let s1 y = (g y >>= return) in f x >>= s1

Now we can use the monadic “law” that states the m >>= return must be equivalent to m:

\x -> let s1 y = g y in f x >>= s1

And, once more, the eta-conversion help us to remove the let, because s1 == g. Thus we get:

(>>.)  :: Monad m => (b -> m c) -> (a -> m b) -> a -> m c
g >>. f = \x -> f x >>= g

This is as good as I was able to get. Since we’re using >>=, I think this is the best we can get (i.e. we can’t generalize to Applicative).

Chaining several iterator-returning functions

Now, I can define a chain function. It takes a list of several a -> m a functions and compose them together (from right to left, as expected):

chain :: Monad m => [a -> m a] -> a -> m a

My first attempt was:

chain :: Monad m => [a -> m a] -> a -> m a
chain []  = return
chain (f:fs) = f >>. (chain fs)

But, then I realized that’s a fold:

chain :: (Foldable l, Monad m) => l (a -> m a) -> a -> m a
chain = foldr (>>.) return

And that completes our incursion in Haskell.

Doing the same in Python

Going from this Haskell definition of chain to Python is quite easy. But we’re not going to work with any possible monad, just lists (iterators, actually).

from functools import reduce

def iter_compose(*fs):
    if len(fs) == 2:
        # optimize the 'lambda x: [x]' for the *usual* case of 2-args.
        return _compose(*fs)
    else:
        return reduce(_compose, fs, lambda x: [x])

def _compose(g, f):
   return lambda x: (z for y in f(x) for z in g(y))

We have included iter_compose() in xoutil 1.9.6 and 2.0.6.

]]> Wed, 01 Aug 2018 00:00:00 -0400 http://mvaled.github.io/blog/html/2018/06/22/a-failing-fail-in-the-monad-list.html http://mvaled.github.io/blog/html/2018/06/22/a-failing-fail-in-the-monad-list.html <![CDATA[A failing fail in the monad list]]> A failing fail in the monad list

Update.

The question was answered in Stack Overflow, and it shows that the title of this post is incorrect: The failing fail is in the function monad.

I’m following the Real World Haskell book. In the chapter about Monads, they create simple example using the list monad compute all pairs of numbers (x, y) that such that x * y == n for a given n.

Their solution is:

multiplyTo :: Int -> [(Int, Int)]
multiplyTo n = do
      x <- [1..n]
      y <- [x..n]
      guarded (x * y == n) $
        return (x, y)

guarded :: Bool -> [a] -> [a]
guarded True xs = xs
guarded False _ = []

But I was wondering if I could restate guarded for any monad. Since fail in the list monad is fail _ = [], I though I could do:

guarded :: Monad m => Bool -> m a -> m a
guarded True = id
guarded False = fail "skipped"

However this actually fails in ghci:

*Main> multiplyTo 24
*** Exception: skipped

I let myself sleep, and this morning I figured there’s actually a small/important difference between guarded False _ = [] and guarded False = fail "..."

The type of guarded False is Monad m => m a -> m a. However the type of fail 'skipped' is just Monad m => m a.

Why does guarded False = fail "skipped" type-checks? That’s an open question (as of the time of writing).

If I define guarded as:

guarded :: Monad m => Bool -> m a -> m a
guarded True xs = xs
guarded False _ = fail "skipped"

Or:

guarded :: Monad m => Bool -> m a -> m a
guarded True = id
guarded False = \s -> fail "skipped"

Both work correctly.

Warning

fail is strongly discouraged to use in real world haskell code.

]]> Fri, 22 Jun 2018 00:00:00 -0400 http://mvaled.github.io/blog/html/2018/01/22/foldl-in-terms-of-foldr.html http://mvaled.github.io/blog/html/2018/01/22/foldl-in-terms-of-foldr.html <![CDATA[Dissecting foldl in terms of foldr]]> Dissecting foldl in terms of foldr

The book ‘Real World Haskell’ provides an implementation of foldl in terms of foldr:

myFoldl:: (b -> a -> b) -> b -> [a] -> b
myFoldl f z xs = foldr step id xs z
   where step x g a = g(f a x)

They ask the reader (me) to try to understand the implementation. They warn it’s not trivial, but I found it interesting. So, I just want to share my line of thought.

The first thing I noticed is that z is not an argument to foldr; you can rewrite the first line as:

(foldr step id xs) z

This makes visible that the result of the foldr is a function that takes z as an argument. By looking a the case foldr step id [], which equates to id, we can see that the type of the foldr’s result is b -> b.

This also implies that the type of step must be a -> (b -> b) -> (b -> b). You can prove that by extracting step:

step1 :: (b -> a -> b) -> a -> (b -> b) -> (b -> b)
step1 f x g a = g(f a x)

myFoldl1 :: (b -> a -> b) -> b -> [a] -> b
myFoldl1 f z xs = foldr (step1 f) id xs z

Use ghci to verify that.

Now comes the funny part of the implementation: step is defined with three arguments

where step x g a = g(f a x)

but foldr only passes the first two, and that’s totally fine; the result will be another function.

You can make this explicit:

myFoldl2 :: (b -> a -> b) -> b -> [a] -> b
myFoldl2 f z xs = foldr step id xs z
   where step x g = \a -> g(f a x)

That’s a beautiful thing about Haskell both definitions of step are actually indistinguishable:

step x g a = g(f a x)

step x g = \a -> g(f a x)

The first one takes an equation-like outlook that I find very elegant in a programming language.

Having all that we can easily follow how the expression myFoldl (+) 0 [1, 2] would proceed:

myFoldl (+) 0 [1, 2]
(foldr step id [1, 2]) 0                  -- by def. of myFoldl
(step 1 (foldr step id [2])) 0            -- by def. of foldr
(step 1 (step 2 (foldr step id []))) 0    -- by def. of foldr
(step 1 (step 2 id)) 0                    -- by def. of foldr for []
(step 1 (\a -> id(a + 2))) 0              -- applying `step 2 id`

(step 1 (\a -> a + 2)) 0                  -- applying id, Haskell might not
                                             do this right now

(\x -> (\a -> a + 2)(x + 1)) 0            -- applying step
(\x -> (x + 1) + 2) 0                     -- applying the inner lambda
(0 + 1) + 2                               -- applying the outer lambda

And that’s (more or less) how myFoldl works.

]]> Mon, 22 Jan 2018 00:00:00 -0500 http://mvaled.github.io/blog/html/2016/09/19/uncurated-notes-about-cycle-js.html http://mvaled.github.io/blog/html/2016/09/19/uncurated-notes-about-cycle-js.html <![CDATA[Non-curated notes about Cycle.js]]> Non-curated notes about Cycle.js

In the past few days I have been reading about Reactive Programming. Mostly about how it’s done with cycle.js. As the title attempts to suggest, this post is by no means an account of well thought ideas, but my first ideas and notes while reading about it.

Cycle.js is attractive, and its appeal spans from two key properties in my opinion:

  • There’s a single type of connector between the components: the streams.

    Note

    Both sources and sinks return streams.

  • There are two distinct type of computational components: the drivers and the the main(sources) function.

    Note

    Intent, view and model functions may be regarded as internal structure of main().

This is appealing because the architecture is simple. You may understand the main points of this architecture in about an hour. You may recall this is actually an evolution (or instance) of the producer-consumer pattern.

There’s a third element in cycle.js: the cycle itself. Which is based on the ‘dialog abstraction’. Actually, this is what caught my attention in the first place. It goes like: your program outputs are the input to an external (possibly human) entity which, in turn, may react and produce more events in your program’s source streams.

Architecturally this is simple and good.

Components

Any Cycle.js app can be reused as a component in a larger Cycle.js app.

—From the Cycle.js documentation

When it comes to designing an application or component you have to decide about the type of events your application may receive and the expected output events it may produce in response.

The previous statement is only true in two cases:

  • Your ‘smaller’ Cycle.js application does not need to interact with other parts of your ‘larger’ application.
  • Your ‘smaller’ Cycle.js exposes its model; or you have “model drivers”.

I think this becomes rather obvious in the 16th part of the Egghead’s video series on Cycle.js. The one when they make the slider component and they need to expose the stream of ‘values’ from the slider.

Note

Actually the documentation states that components expose, somehow, its state.

Streams to the extreme (and localization)

The Egghead’s series exposes how Cycle.js became to be and you can, therefore, see the evolution it is. Is a short video series worth watching: right to the point without many detours.

There’s a video when they needed to make a component out of the Height and Weight sliders, and they needed to pass parameters like the label, min value and max value. I thought they were going to use old plain closures:

function Slider(sources, props) {
    const label = props.label;
    // etc...
    return main(sources);
}

It came as a surprise that they put those parameters into a props source to the main() function of the component.

At first I thought that was really far fetched, but after thinking about it a bit more I think is can be really useful.

I started to think about an application with just a slider and a language selector:

Height: 6 feet   <-----o-->

Choose language: [ENG]

The slider component is the one we see in the video series. The language selector is a selection component and the expected behavior is that when I change the language the entire application changes to the new language.

My first thought is that the language selector gets is value from a driver (which I don’t know if it already exists) that deals with localization. Let’s say you can obtain such a driver like this:

cycle.run(main, {
   // etc...
   locale: makeLocaleDriver({languages: ['en_US', 'es_ES'], ...})
})

The driver would let you respond to changes in translations:

const label$ = sources.locale.select('Height');

Or combining with another mapping from another stream so that the props stream remains almost unchanged:

const label$ = props$.map(p => sources.locale.select(p.label));

However, after revisiting that last idea, I noticed that it doesn’t work. A change in in the locale does not trigger any event in the props$ stream. Assuming that locale.current$ is a stream of localization object, this may work:

const label$ = props$.combine(sources.locale.current$)
                     .map((label, locale) => locale.gettext(label));

The thing gets a little bit more tricky when it comes to changing the units: feet vs meter, etc… I’ve been thinking about it for a bit. The most problematic issue is that state is not clearly owned unless we introduce a kind of quantity for which the unit of measure is explicit:

run(main, {
    props: xs.of({
       value: new Quantity(175, Unit.Length.cm),
    })
})

However this may seem a bit overreaching for a single value that only needs to be in between two boundaries (slider).

This is, IMO the breaking point: If I really need to manage units on my application and those need to be fully localized, my components might be regarded as over-engineered for other apps. My only hope is that a simple slider, without any knowledge of units, might be wrapped inside a FullyLocalizedSlider for that purpose.

Open questions

Most of the ideas exposed above are not battle tested. I happen to be evaluating whether I could use Cycle.js inside Odoo to develop some widgets that require almost real-times updates, and the stream interface is thus quite natural.

There are challenges about integrating my components with the rest of the application, and being an application that must display at least three languages I need to think on advance about the problems I would face.

]]> Mon, 19 Sep 2016 00:00:00 -0400 http://mvaled.github.io/blog/html/2015/09/12/odoo-bus-implementation.html http://mvaled.github.io/blog/html/2015/09/12/odoo-bus-implementation.html <![CDATA[The bus implementation in Odoo and notification systems in the Web]]> The bus implementation in Odoo and notification systems in the Web

Delayed post.

This post has been delayed for a very, very long time. If I trust the logs in my blog’s git repository the post is “finished” since Sat Sep 12, 2015. Well, that’s not true, and I must really finish it now.

As it happens, in the meantime, I was not twiddling my fingers and I made two “advances” about the bus implementation:

I commented on my post about our migration to Odoo that our users open many tabs to the same Odoo instance. When the chat is active this means an open connection for each tab and, since a chat is interactive in nature, all but one of these connections are not actually required [1].

A while ago I reported that switching from a threaded based deployment to a pre-forked processes one with a gevent based process for long polling connections, actually spiked the concurrency issues we were witnessing in our DB. Our DB complained about being unable to serialize concurrent transactions. By inspecting the logs we noticed that almost always this had to do with an update to the table im_chat_presence, that holds the status of users in the chat.

I thought the forked model would diminish the chance of collisions because (I thought) Odoo would use a single DB connection per worker process and another one for the gevent worker that would simply serialize all the chat related queries.

This spike was against all rationale I could think of, but honestly I didn’t check my assumptions by reading the code. They might just be the wrong assumptions in the first place, and I needed to really read the code to see what was really happening.

After restoring the threaded deployment the concurrency issues dropped again. They didn’t get to the same levels they were before, but, at the same time, we were introducing other modules (and the related staff), so there are more open tabs… Afterwards we went back to the process model, and the issues spiked again, however in a smaller amount.

Several weeks later I deployed a patch to the Odoo bus implementation that simply reduced the frequency of polling when the page is hidden. The concurrency issues were now mostly gone: from about 3k a day to no more than 50.

We still had unanswered questions regarding the DB connection management in Odoo. There were suspicious signs. For instance reducing the db_maxconn option to 8 or less made Odoo start complaining about the Connection Pool being full very often. We already knew that:

  • For each cron worker at least 2 connections are used: one holds the cron job locked while the other is used by the job itself.
  • The bus (used for the chat) also uses another connection to the DB “postgres”: Notifications and wake-ups of bus-related events use this connection.

The most suspicious sign, however, was that with a pool size of 32, this issue came from time to time.

Time went by and the issue remained at an acceptable (for us) level of few dozens per day. You may say it’s too much, but, bear with me, it’s reasonable for our hardware and user base.

Just a few days ago, we realized that issues coming from our gevent-based process were not being properly reported to our Sentry. We fixed the issue and observed that we’re getting about 2.5k DB collisions per day reports again. And once more the query the table where most of collisions happen is ‘im_chat_presence’.

As a workaround we simply reduced the update rate for that table. But a better solutions needed to be found.

Cross-tab polling

By looking better our users’ usage pattern, we discovered that users switch from tab to tab regularly and thus the polling is still high. So we went down the path of figuring out a way to have a single polling connection even when many were opened.

I remembered the Local Storage API and went down to write a simple proof of concept of listening events of the localStorage from different tabs. The proof of concept yielded good results, and so I turned to Google and search for “cross-tab communication” and found the tabex library.

After that, replacing the implementation of the bus for a new one that was based on a exchanged between tabs was not very hard. These are the commits done in our branch (in reverse topological order as in git log):

0555434 * bus: Clarify the channel delay in polls.
d10a1b5 * bus: Checks for the right state.
af4a6dc * bus: Add language tag in the documentation.
636490b * im_chat: Revert "Reduce the presure on im_chat_presence table."
eda7638 * bus: Don't notify if not in polling state.
4d09ac6 * web_celery: Use a dedicated bus exchange for jobs.
84239e3 * im_chat: Beep only in the master tab.
2cb27da * Improve the bus to allow tab synchronization.

The file addons/bus/static/src/js/bus.rst contains the new design explained.

A few weeks ago I did the first merge our 8.0 changes into the 9.0 branch. On the process I realized that Odoo 9.0’s bus was improved and now contains the a cross-tab communication feature.

So probably I end up backporting their implementation into 8.0.

Notes

[1]Of course it allows the web app to change the window’s title when then user has an unread message, but in our case this notification is also unneeded, because the message is actually being read in another tab.
]]> Sat, 12 Sep 2015 00:00:00 -0400 http://mvaled.github.io/blog/html/2015/08/04/repost-the-productivity-of-tomorrow-trumps-that-of-today.html http://mvaled.github.io/blog/html/2015/08/04/repost-the-productivity-of-tomorrow-trumps-that-of-today.html <![CDATA[The productivity of tomorrow trumps that of today]]> The productivity of tomorrow trumps that of today

Editorial note

I’m republishing this post from my old blog (which is no longer online). It’s published as it were when I first made it online with only minor changes. But I think I stand by this still.

Hope you’ll enjoy the reading.

That’s probably harsh, but I think it is absolutely right. Doing crappy software today to be more productive today will make you less productive tomorrow. It’s this simple. And that’s cumulative too; meaning that if you neglect your future productivity, it will be slowly diminishing until a point of negative competitive disadvantage where you’ll find yourself struggling to keep up instead of innovating.

And it’s so damn exhausting to explain why…

Software complexity does not come from the tools, but from the mental framework required (and imposed at times) to understand it. So don’t ever think measuring Cyclomatic Complexity (CC) and other similar metrics will yield something close to the true measure of the quality of your code.

There are only two hard things in Computer Science: cache invalidation and naming things.

—Phil Karlton

def _check(l):
    if len(l) <= 1:
       return l
    l1, l2 = [], []
    m = l[0]
    for x in l[1:]:
       if x <=m :
          l1.append(x)
       else:
          l2.append(x)
    return _check(l1) + [m] +_check(l2)

This code has a nice CC of 4 which is very nice; yet it will take you at least a minute to figure out what it does. If only I had chosen to name the function quicksort…

>>> _check([84, 95, 89, 4, 77, 24, 95, 86, 70, 16])
[4, 16, 24, 70, 77, 84, 86, 89, 95, 95]

A quiz in less than 5 seconds: What does it mean in OpenERP’s ORM the following lines of code?

group.write({'implied_ids': [(3, implied_group.id)]})

This line of code has a CC of 1: as good as it gets, isn’t it? But it’s so darn difficult to read that unless you have your brain wired up to see “forget the link between this group and the implied_group”… To be fair there is someone out there in the OpenERP universe that cared a bit about this:

# file addons/base/ir/ir_fields.py

CREATE = lambda values: (0, False, values)
UPDATE = lambda id, values: (1, id, values)
DELETE = lambda id: (2, id, False)
FORGET = lambda id: (3, id, False)
LINK_TO = lambda id: (4, id, False)
DELETE_ALL = lambda: (5, False, False)
REPLACE_WITH = lambda ids: (6, False, ids)

But no one else is using it!

And, yes, it’s exhausting going over this.

]]> Tue, 04 Aug 2015 00:00:00 -0400 http://mvaled.github.io/blog/html/2015/05/26/odoo-at-last.html http://mvaled.github.io/blog/html/2015/05/26/odoo-at-last.html <![CDATA[Odoo at last!]]>

Odoo at last!

Updated 2015-06-03

The menu decomposition described below was actually an error in a custom addon and not related with migration process.

While still using OpenERP, we backported the partner merge feature from Odoo and made it work in OpenERP. We even added a fuzzy match feature so that minor typos were not missed. Now in Odoo it seems our modifications messed things up.

After several months, we have finally moved from OpenERP to Odoo. It took several rounds of testing, finding bugs and incompatibilities in our own modules and fixing them.

A summary of the process

The OpenUpgrade project proved to be very complete for us –it had almost everything we needed:

  • The Accounting and Project modules were done.
  • The CRM module was not yet integrated into the mainstream repository but a fork by Stefan Rijnhart had a branch that proved very complete for our needs.

We began by merging Stefan’s branch into our own clone of the repository. We came into a small issue and fixed it. Afterwards, some of our recurrent events were missing the timezone and the migration failed. For that we created the branch 8.0-valid-rrules and it did the job after a couple of tweeks.

The branch we actually used for the migration is merchise-8.0. It includes several commits we had to make for our customs addons to work. Those commits are probably too local to be merged back into the upstream. However, we have published them should anyone finds them useful.

The menu decomposition

The biggest issue we’ve encountered so far is that after several hours of operation all the menus stopped working: You clicked the “Leads” menu item and you got, say, the list of Many2Many fields. (???!!!)

A first inspection showed that all menu items in the HTML were somehow related to the same action id. Quickly, we truncated the ir_ui_menu table and its associated ir_model_data rows, upgraded the DB and all menus came to life once more.

The day after, the same problem happened. Deeper inspection of both the code and the DB showed the problem lied in the ir_values table. That table relates menu items with actions. The rows containing this relation showed nonsense: almost all the menu items were related to “res_partner,20”. And in the UI this was interpreted to be pointing to the action with id 20.

Comparing the table before and after the upgrade showed no problem there. So this issue had appeared after the migration itself.

Our previous method of truncating ir_ui_menu didn’t fix this issue cause the ir_values rows remained there crippling over and over again. Besides, truncating ir_ui_menu had the unintentional, bad side-effect of removing every Mail list in our DB.

We’re still on the track about how the ir.values were messed up in the first place.

The magical 6 tabs

Our users started to report slowness after a couple of hours of working. First, we noticed that the call to load_needaction was taking too long and find out this call asked for too much it did not need, we provided a fix and some loose benchmarks. Though this patch needs to be completed, the slowness was solved… for a moment.

Some users kept reporting the Odoo interface was freezing all the time, even inviting them to go out and drink a cup of coffee… But other users were happy about it after the patch. Closer inspection showed many of the AJAX calls were blocking waiting for a socket to be available.

It turned out these users had created a habit of opening several tabs when they worked with OpenERP. They kept this kind of behavior, but a couple of facts worked against them:

  • We have installed the Odoo chat. Many of our users are geographically spread, the chat provided an instant and cheaper communication path.

    This feature was, by far, one the reason we moved to Odoo in the first place.

    On technical side, the chat works by long polling the server. In our server this connection stays open for about 50s before being dropped (only to be reestablished a moment later).

    This means: a tab has always at least a connection open to the server.

  • As explained by Ilya Grigorik in High Performance Networking in Chrome, browsers make only so many connections to the same combination of scheme, host and port.

    In Chrome this number is 6. In Firefox this can be tuned up or down by tweaking the network.http.max-persistent-connections-per-server parameter, but the default is 6.

Combining these, it means that after opening about 6 tabs every AJAX call has to wait up to 50s to be able to even reach the server.

We opted for raising this number in Firefox and to warn users about this. Our user base is small enough and the server has yet to achieve any noticeable load.

Epilogue

That’s what our migration to Odoo story has gone so far. A little bit stressful at times, specially when the menus went south for the second time, but I think we have crossed the yellow line now. Should anything comes I’ll report it.

I’d love to hear (or read) about others, so I can learn from their experience.

]]> Tue, 26 May 2015 00:00:00 -0400 http://mvaled.github.io/blog/html/2015/04/08/rememoring-microservices.html http://mvaled.github.io/blog/html/2015/04/08/rememoring-microservices.html <![CDATA[Memento: microservices]]>

Memento: microservices

Lately there’s been a raise of the debate about Microservices architectures. Reading about the stuff I realized I did once architect an application called Pyxel based on the same principles, though different.

Pyxel reached only its first prototype and was never actually deployed. I’d say it wasn’t given the chance to prove itself. So I argue it was technically sound and, although only a prototype and thus needing improvements, it would have meet its stated requirements [1].

I haven’t been given permission to freely distribute the source code. But the architecture was published in several public events, so I can freely describe it here.

Pyxel

Pyxel’s goal was simple:

Be a shared repository of photographs for the news media that wish to collaborate.

Pyxel required that images were tagged with IPTC metadata before uploading. Upon upload, the image went through several processes for extracting the metadata, and for producing web-friendly versions [2] for several standard dimensions before being published.

On the technical side we have also very hard constraints:

  • the system would have to be highly accessible.
  • it should run on spare machines; introducing a new machine to replace an old one should be possible with minimal interruption.

This constraints were the main driver that lead us to design what we called a distributed system. Each major component was designed to run in isolation of the rest.

Pyxel was split in several processes:

  • An image queue holding new images. This was kind of a FIFO queue over the network.
  • A catalog holding indexes for retrieving images based of queries. Queries were defined a syntax allowing from very simple queries aided with pattern recognition (PR) of names and dates, to very specific queries that make the PR mostly unused.
  • A “feature” index. This basically allowed to find similar images. There were two indexes, one based on Hu moments and the other based on wavelets.
  • Several image processing nodes, that take images from the queue and:
    • Extract IPTC metadata and put it the catalog
    • Extract image features to allow detection of similar images.
    • Produce “web versions” for the image. We always kept the original file, but to keep the design of the prototype simple there was an “as-is web version” that simply returned the same file as the input.
  • A web application that communicates with the index and the catalogs.
  • An FTP server that injects images to the queue using a FUSE mount point (drag-and-drop to the browser wasn’t common yet).

That were the main components and all of them ran independently. Of course for some services to perform at 100%, they required others to be working. The web application would not be able to perform a search if the catalog was down, or to upload new images if the queue was out of reach.

The principles stand

Microservices are “a take” on the same old issue about software composition. Conceptually Parnas still rules: must do modules. Names have changed and several rules to build better modules (components, services, etc) have been provided.

Exposition of the modules as services have also been promoted.

In the web, the modules have drifted away from the server into the client.

Microservices and Pyxel

Pyxel was not actually a microservice architecture as it is understood now. Deep down it used a central name-server that matched services names to network endpoints. We used Pyro for this prototype.

We did recognize that this was an accidental choice, a matter of convenience to have the prototype working as soon as possible.

We chose Pyro to test our prototype not to keep it forever if it performed poorly. And though our first tests were positive there was some slowness.

To accommodate for a possible replacement of Pyro we chose to provide our own “idiom” (in Python) to connect and communicate with services:

with service('pyxel.queue') as queue
    queue.method(*arguments)

That pattern was designed to have an appearance as simple as RPC. Later we found that RPC was under fire, but the project was being shut down, so we didn’t fix that.

Under current views Pyxel needs several changes. I can think of a couple:

  • Probably exposing the queue to the client-side of the web application instead sending the images first to the server-side of the web application.
  • Extract the user registration, authentication and authorization into a service.

Notes

[1]Actually one of the main causes of Pyxel being a failure was I could not make all participants agreed on the requirements.
[2]Current retina displays would make that thumbnails look like icons and bandwidth is also more likely to allow very big pictures to be friendly.
]]> Wed, 08 Apr 2015 00:00:00 -0400 http://mvaled.github.io/blog/html/2015/02/18/greenlet-local-python-modules-an-experiment.html http://mvaled.github.io/blog/html/2015/02/18/greenlet-local-python-modules-an-experiment.html <![CDATA[Hot-swapping Python modules. An experiment.]]> Hot-swapping Python modules. An experiment.

A new project I’m involved with will probably require dozens of servers running several thousands greenlets each. Top-level greenlets represent jobs and their children will be individual tasks those greenlets are coordinating/supervising.

This model, however prototypical, resembles that of the OTP in Erlang. A greenlet may be either a supervisor or a worker.

But there’s something missing in our platform that Erlang do have and that might yield huge benefits. You can change your Erlang code while the program is running.

Modulets. The idea

I asked myself if I could devise an import mechanism that would allow to update a Python module in a way that already-running greenlets stay unaffected but newly created ones use the new code.

To exemplify, let’s say a typical tasks is:

def receive_confirmation(message, who):
   from jobs.util.email import send_email
   from jobs.util.email import wait_reply
   # Assume both send_email and wait_reply switch away from the current
   # greenlet and only switch back after they are done.
   message = send_email(message, who)
   res = wait_reply(message)
   return res  # this will be sent to the parent greenlet

The servers start and hundreds of this task are launched in different jobs. Many of them are idle waiting for their replies. Users are happily getting their confirmation emails and replying to them (or ignoring them).

However, we start receiving lot of bounces in the postmaster inbox. Some users have entered a wrong email address. A change is in order.

In response, we change our implementation of send_email so that it does VERP to know which recipients’ address are bouncing, and also create a new job that involves receiving confirmation email when a new user registers.

We’d love to simply update our jobs.util package and be done with it like this:

$ source server-virtual-env/bin/activate
$ pip install -U jobs.util -i https://my.private.server/

New jobs will pick up the new version and the older jobs will keep working as if nothing would have changed.

That would be really nice. Such a hot-swap of Python modules per job is what I’m calling a “modulet”.

Currently I have a “working” yet very experimental and undertested implementation of such a mechanism in our experimental modulets branch in xoutil.

Modulets in xoutil

The current implementation is a very early proof of concept and not something you’d like to put outside your playground.

The file test_modulet.py is a simple script you may run to see it working. It simply creates a temporary module magic_module that has the get_magic function. This function returns a single value.

The test launches three greenlets that simply call the get_magic function and asserts it returns the “right” magic value. Between launches the module gets updated to return a different magic value, which is passed as an argument to the newly created greenlet.

A single run will print something like:

$ python test_modulet.py
Beginning with 3 in /tmp/tmp1d4rK5
Isolating 'magic_module' as '<greenlet.greenlet object at 0x7f21f4e8daf0>.magic_module'
Isolating 'magic_module' as '<greenlet.greenlet object at 0x7f21f4e8da50>.magic_module'
Isolating 'magic_module' as '<greenlet.greenlet object at 0x7f21f4daa7d0>.magic_module'
Passed. I have the right magic number 1002
Passed. I have the right magic number 1001
Passed. I have the right magic number 1000

If you comment the bootstrapping of modulets, then you’ll get something like:

$ python test_modulet.py
Beginning with 3 in /tmp/tmpeI1oYA
Wrong magic number
Traceback (most recent call last):
  File "test_modulet.py", line 49, in rootprog
    g.switch()
  File "test_modulet.py", line 31, in prog
    assert res == magic, "Expected %d but got %d." % (magic, res)
AssertionError: Expected 1002 but got 1000.
Wrong magic number
Traceback (most recent call last):
  File "test_modulet.py", line 49, in rootprog
    g.switch()
  File "test_modulet.py", line 31, in prog
    assert res == magic, "Expected %d but got %d." % (magic, res)
AssertionError: Expected 1001 but got 1000.
Passed. I have the right magic number 1000

Future work

Since we are at the very early stages of this project is not easy to predict if we’ll keep modulets in our platform. Probably a celery based solution be enough.

If we were to keep it, there are several things to improve:

  • The current mechanism pollutes the sys.modules with a copy of a module per top-level greenlet.

    In the current state, this is an ever-growing pile of modules that never erases those that are no longer used.

    This needs to be changed in several ways:

    The namespace we use to masquerade the modules need not be (and should not be) the repr of the greenlet object.

    For the purposes of isolating different versions of the same code we can either use the timestamp of the files, the version of the distribution, etc…

    Running a diesel server will quickly eat all your RAM unless this is changed.

    When a greenlet dies the only one informed is its parent. But we certainly don’t want jobs to mess with sys.modules to clean our own mess.

    This poses a challenge of its own and may be delegated outside xoutil itself.

    That being said, it’s likely that the calculation of the current namespace and how to dispose of unused modules will be extensions points of modulets.

  • Currently we have a black-list of modules that will never be isolated.

    Changes in those modules will required a restart to be noticed. Those modules are platform-level. They include xoutil itself, greenlet and the entire standard library (which is not expected to change unless you change Python).

    We can also allow white-listing. Both ways are on the table.

    The white-list imposes more explicit architecture of your platform since it requires throughout revision of which modules you’re willing to update on the run.

    Access to both lists will be a public API of the Modulet Manager. I can envision a remote-control console you’ll use to include a new module in the white-list. But that will be an application of the modulet API and included in the box.

]]> Wed, 18 Feb 2015 00:00:00 -0500 http://mvaled.github.io/blog/html/2014/12/27/lately.html http://mvaled.github.io/blog/html/2014/12/27/lately.html <![CDATA[What I’ve been doing lately]]> What I’ve been doing lately

Lately my work has been mainly driven by the need to efficiently and smoothly deploy the Accounting module of OpenERP in my employer enterprise. But at the same time I’m responsible of several development and deployments in the enterprise.

The accounting-related tasks have made me to study the accounting module, but also accounting in general from fundamental principles to advanced topics. I currently own a collection of 9 books about accounting. I still have more studying to do, but now I’m versed enough to the point of being a (very junior) accounting consultant to the CEO. This, of course, should yield future benefits since I’m planning to start a business.

On the technical side, the last week I was doing a final verification of the module so that the (true) accountant of the enterprise finally migrates to OpenERP: Taking our QuickBooks files and, with Pentaho Kettle, migrating the accounting information to OpenERP, and comparing the legal reports. This has finally yielded good results and we’re doing an opening entry to continue accounting in OpenERP (the enterprise has an accounting period the starts on Oct, 1st each year).

At the same time a new server for OpenERP and a deployment based on Buildout was another one of my tasks. There’s still much to be done and I’m wondering if Ansible would be a good choice to actually manage the deployment since it involve setting up Postfix as the Mail Transfer Agent interfacing OpenERP for emails. This means that besides the accounting books I have reviewed several email related RFCs (2822, 2821 and 2476 being those I’ve read the most) and the Postfix documentation.

To fill the gaps, or more precisely: to take breaks from these topics I’m reading a couple of books and articles:

Things left in the attic

At the beginning of 2014 I expected I could dedicate myself to finish xotl.ql by August. This was, however, not possible. I still keep my eye on the topic from time to time, but this needs a lot of time to actually make a breakthrough.

]]> Sat, 27 Dec 2014 00:00:00 -0500 http://mvaled.github.io/blog/html/2014/12/09/apples-and-oranges.html http://mvaled.github.io/blog/html/2014/12/09/apples-and-oranges.html <![CDATA[Apples and Oranges. How books can mislead our thinking process.]]> Apples and Oranges. How books can mislead our thinking process.

I’ve just began to read the book “Twisted Network Programming”… Twisted is a popular framework for networking programming. However, I’d probably skim most of the Book. I’m sorry… But as it happens, in the second page of the second chapter (the one that I began with) they show an illustration on “Comparing single-threaded, multithreaded, and event-driven program flow”.

Hum… apples and oranges. I mean the thread model of a program will have some impact on how events are dealt with, but are they truly comparable or the sole responsibles for the program flow?

Diesel is usually ran in a single thread and so is Tornado. You may, of course, run programs written with them in multiple processes (or threads) to avoid contention when the volume of clients (events) is too high.

Even with a single thread most of the time the system is waiting for the IO buffers to have data, so they can be classified as event-driven: They actually response to events.

But what really bothers about this illustration is not that it compare two thing that could be (and should be made) complementary. What bothers me is that the picture tries to fool me by showing that event-driven programs don’t have any “gaps” of idle time between tasks, while the threaded models do. And that’s simply dishonest.

Probably this is not what the authors meant, and I’m probably going prejudiced just because of this wrong picture. However, I go to a book like this not for pleasure but for information. Finding a piece of data that is not trustworthy raises a flag and I start to read with a magnifying glass… And that’s not fun either.

Don’t expect me to finish this book.

Update of a couple of hours later…

A few lines below the illustration of matters, they actually say:

“The event-driven version of the program interleaves the execution of the three tasks, but in a single thread of control.”

So, let’s give it another try.

]]> Tue, 09 Dec 2014 00:00:00 -0500 http://mvaled.github.io/blog/html/2014/08/28/reviewing-openerp-8-0.html http://mvaled.github.io/blog/html/2014/08/28/reviewing-openerp-8-0.html <![CDATA[Reviewing Odoo 8.0]]> Reviewing Odoo 8.0

This is the first of a series of blog posts related to the next OpenERP 8.0 (or should I say Odoo 8.0). This first post will be very “seeking for orientation” on major new features and changes. Don’t expect anything to be deeply commented or analyzed.

I will probably devote several posts for changes in some modules, but in this one I’ll just write about things that can be immediately seen when you first install it.

Installing Odoo 8.0

I’m working with a clone from the now official github.com odoo repository. Of course I’m using the branch “8.0” for this review.

I prepared a virtual environment and installed odoo there like this:

$ python setup.py develop

The first I noticed is that several new requirements are in place, and the pyPdf requirement had been forgotten.

Second thing noticed is that unlike the “7.0” you don’t need to run the openerp-server script passing the --addons-path=./addons. In 8.0 they include this path automatically. So running ./openerp-server is enough.

Also the python setup.py sdist command includes this path as well, so a properly built distribution from source seems more doable. I haven’t tried this yet, but I will, cause that was for sure something missing in 7.0. It’s quite suspicious that the tarball is too small (around 13MB) though.

New addons

Instant Messaging

Once I ran the server I installed a demo DB to see what’s new. The first thing that captured my attention was an “Instant Messaging” module. I gave it a try and it seemed to work out-of-box.

It uses the quite old long polling technique instead of the brand new HTML5 Server-Sent Events (SSE) stuff. Why?

My first guess was that multi-process deployments would need to pass messages around between the HTTP processes. That requires a kind of Inter-Processes Communication (IPC). Popular choices for achieving this in web applications are Redis and RabbitMQ, but then you’d need to setup either for this to work.

After thinking in this a bit more (actually the thinking about this occurred at the same time I was writing this post) I realized that the current multi-process deployments would still need this IPC to properly work. So I ran ./openerp-server --workers=2 and tested it again. The chat stopped to work and the following error was logged from time to time:

2014-08-27 22:53:48,972 17858 ERROR o8 openerp.http: Exception during JSON request handling.
Traceback (most recent call last):
  File "/<..>/odoo/openerp/http.py", line 476, in _handle_exception
    return super(JsonRequest, self)._handle_exception(exception)
  File "/<..>/odoo/openerp/http.py", line 495, in dispatch
    result = self._call_function(**self.params)
  File "/<..>/odoo/openerp/http.py", line 311, in _call_function
    return checked_call(self.db, *args, **kwargs)
  File "/<..>/odoo/openerp/service/model.py", line 113, in wrapper
    return f(dbname, *args, **kwargs)
  File "/<..>/odoo/openerp/http.py", line 308, in checked_call
    return self.endpoint(*a, **kw)
  File "/<..>/odoo/openerp/http.py", line 685, in __call__
    return self.method(*args, **kw)
  File "/<..>/odoo/openerp/http.py", line 360, in response_wrap
    response = f(*args, **kw)
  File "/<..>/odoo/addons/bus/bus.py", line 188, in poll
    raise Exception("bus.Bus unavailable")
Exception: bus.Bus unavailable

The code explains it all. This is the method ImDispatch.start() in file addons/bus/bus.py:

def start(self):
    if openerp.evented:
        # gevent mode
        import gevent
        self.Event = gevent.event.Event
        gevent.spawn(self.run)
    elif openerp.multi_process:
        # disabled in prefork mode
        return
    else:
        # threaded mode
        self.Event = threading.Event
        t = threading.Thread(name="%s.Bus" % __name__, target=self.run)
        t.daemon = True
        t.start()
    return self

The highlighted line says this won’t work in a multi-process deployment. How do you get to use the evented mode, I don’t know. Probably that’s the default now.

So I need to review this feature more closely before going to production. It’s a nice addition though.

I found they have now a GeventServer for long polling connections. And the implementation of the Instant Messaging Bus (bus.py) can be easily adapted for desktop-like notifications, updating your message inbox, and many other features that would benefit from this.

Messaging has gone a bit different

At this point I tried to send a message from a user to another (to test if the inbox was updated real-time) and realized that the Messaging addons has lost the “compose a new message” that was previously accessible from the inbox, let me show you a picture (7.0 on the right, 8.0 on the left):

../../../_images/odoo-vs-openerp.png

Odoo is missing the “Compose new message” button.

This commit has some explanation:

commit 0714b231646bb439b121a6aaa43df32fedcb5e6e
Author: Thibault Delavallée <tde@openerp.com>
Date:   Wed Aug 13 14:35:25 2014 +0200

[FIX] mail: when having only mail installed, do not show any 'share with my
followers' compose box. This comes only with hr, for the inbox. This was
probably forgotten when updating the mailboxes hr-goal and hr-related
gamification / chatter stuff.

I then installed the “Employee directory” addon and the “Share with my followers” box appeared.

More digging in the logs reveals the following:

commit e6f8666b521fe8c2522d6e94c0c3def54a5f73ed
Merge: d57a97d bf3d4a7
Author: Amit Vora <avo@tinyerp.com>
Date:   Thu Apr 17 11:41:33 2014 +0200

[MERGE] [IMP] mail: Inbox usability improvements :
- notficiation_email_send field, renamed into notify_email, has now 2 values: always or never, in
order to ease the choice and simplify options.
- inbox: removed 'compose a new messages or write to my followers', because those 2 options are
already available. The first one is accessible using the top-right email icon, the second one
is accessible with the 'write to my followers' text box alread present in the inbox.

However I don’t see the mentioned “top-right email icon”. In fact, that icon is present in 7.0, but not in 8.0. More digging:

commit 5209fbc7ed9fcad966ab064654a8a8697142be42
Author: Antony Lesuisse <al@openerp.com>
Date:   Mon Jun 30 01:51:40 2014 +0200

[REM] useless icon send a message

The action is available from the wall.

Got you! So, Antony removed the icon cause he thought it was the wall, but Amit had removed it from the wall cause because of the icon. These things happen…

After reverting this commit, the icon is reestablished. But, then I realized that the feature, however hidden, is actually there: If you click on the “Share with …” and instead of writing your message there, click on the “expand” icon, then you get the same as pressing the (now missing) email icon. I think that the icon is required, though, since now the “Share…” box is not shown until HR is installed, and the email icon allows to send email to outsiders. My current employer is willing to remove email in favor of this messaging module. This may hold him back.

I’m raising my hand to vote for the rescue of (at least) the icon. Redundancy is not always bad when it comes to user interface.

This enough for this post. I’ll keep looking at Odoo and I’ll write about it.

]]> Thu, 28 Aug 2014 00:00:00 -0400 http://mvaled.github.io/blog/html/2014/07/02/integration-v-automation.html http://mvaled.github.io/blog/html/2014/07/02/integration-v-automation.html <![CDATA[Integration v. Automation]]> Integration v. Automation

Editorial note

I’m republishing this post from my old blog. It’s published as it were when I first made it online with only minor changes. I’m planning a follow up on the subject and more post about my experiences with Odoo (OpenERP). That’s I deem this repost needed.

Hope you’ll enjoy the reading.

Being an OpenERP bee these days has lead me to think a bit about integration, how does it happens inside OpenERP, its architecture, assumptions and other stuff. In a first iteration I used to think about integration in terms of integrating two (or more) OpenERP addons, for instance, integrating Sales with Accounting. At a very programming-oriented level this reduces, in the current state of OpenERP, to integrate their modules. But later, I found out that this approach is fundamentally flawed and causes many quirks.

This post starts with a tale about one of the many issues we have encounter when deploying OpenERP in an small enterprise. Then I will argue about what integration means. Lastly, I will turn to automation as a mean for integration and not the only way to do it.

The Integration tale

I want to start with a simple and very real [1] use case: Bob is a salesperson processing a CRM lead.

The story beings when Bob opens the lead and clicks on the Edit button and the first thing he tries to do is to fill the client’s contact info because it’s a new client. He clicks on the edit button besides the “Client” field which opens a window. He fills the data and clicks “save”, but suddenly it fails saying that Bob (a salesperson) should have filled the “Account payable” and “Account receivable” fields. So Bob says:

Oh, boy! What did I do wrong?. Let’s try again… Nope… Uh. What should I do?

Hey! Peter. Do you know what’s happening here? This won’t let me enter this client’s contact data, it keeps telling me that these accounts are wrong? What the heck are these accounts anyway?

And Peter has no idea, so he calls Sally who’s a very pretty girl that probably does not have a clue either about what’s happening but it’s so nice to have a reason to approach her… Surprisingly Sally seems to be only one who actually paid attention to the information meeting that morning and she says:

Ah! This probably has to do with the Accounting module being installed. They say they will do that today… Let’s ask Stephanie.

And Stephanie realizes what’s happening and fix it somehow. An hour later Bob can resume his work and finish processing his CRM lead.

The end.

So, what is wrong with this story?

Computer problems.

Computer Problems. Taken from xkcb.

Depiction of wrongness

In my opinion there are two things that are clearly wrong with this:

  1. Requiring data when it’s not required. When you’re in a CRM use case requiring accounting-related seems really overstepping.

  2. Not being able to issue a request for completion. Bob should have been able to keep working and after the new client was saved a completion request should have been sent to those being able to complete the data.

    That completion request might just get ignored, but other use cases (like submit an invoice to that client) would actually require the data and thus the cycle would be closed after all.

Some would think that this is just a “configuration problem” as we will discuss below. But finding that solution took me several dives into the code and re-parsing of the documentation, so I’ll try to walk you through the process.

The issue. The making off

Warning

Technical stuff ahead. Wearing hardhats is mandatory.

By installing the “Accounting & Finance” addon [2] you touch many parts of your system besides installing new stuff. In fact, the module forces the installation of many other modules, including product [3].

It also modifies the "res.partner" model from the base addon. You may think about the "res.partner" model as an attempt to merge the contact information for both clients, suppliers and employees into a single entity [4]. It has many fields that comprise name, job position, contact information, etc…

When you install the account addon, the "res.partner" gets appended a whole bunch of other fields, including the said “account payable” and “account receivable” fields, which are also marked as mandatory:

'property_account_payable': fields.property(
    'account.account',
    type='many2one',
    relation='account.account',
    string="Account Payable",
    view_load=True,
    domain="[('type', '=', 'payable')]",
    help="This account will be used instead of the default one as the payable account for the current partner",
    required=True),
'property_account_receivable': fields.property(
    'account.account',
    type='many2one',
    relation='account.account',
    string="Account Receivable",
    view_load=True,
    domain="[('type', '=', 'receivable')]",
    help="This account will be used instead of the default one as the receivable account for the current partner",
    required=True),

This are not actual fields of the model "res.partner", but properties, which are “special”.

On using properties

Properties are of course related to the “solution” for the problem described above. But the solution is well hidden under the title of Database setup in the OpenERP Book. That’s the reason I’m using this case to open the OpenERP corner. If you deploy CRM before accounting [1] you’d probably find no interest in reading a topic called “Database setup”… you have set your database up already, haven’t you?

You should notice that both Account receivable and Account payable are, in fact, properties (i.e defined via fields.property). This actually means that those fields take their default values from a global configuration.

Those values were not properly set in our case cause there were no localized account chart that applied to our enterprise. We have to create all the accounts by hand, and yes, we missed (didn’t know) that we have to create those properties.

Our problem is solved by defining those properties in the configuration menu.

However this workaround is very unsatisfying:

  • It involves the administrator of the system because he’s the one that has access to the “Configuration Parameters”. AFAIK, the accountant himself/herself cannot change the defaults, unless you bestow all the powers on him/her.
  • It does not resolve the integration problem others might present in the future. Integration is harder that having some default values. For instance, Cuban accounting norms establish more than 10 accounts for receivables with empty slots for more if needed. They require to have different accounts (receivable and/or payable) for bills to/from people (B2C) separated from those bills to/from other enterprises (B2B), and also different accounts for long-term and short-terms bills. The last case cannot be decided by just looking at the client or supplier, more information about the economic fact is needed.
  • It does not actually resolve the current integration problem since the accountant needs to make sure the “Account Receivable” is the correct one for the client and he’s not notified when salesman Bob creates a new partner. So, what really happens is that the accountant needs to review journal entries and before posting them, and change the account if needed.

Integration v. Automation

An ERP should simplify things by integrating business areas, shouldn’t it? That’s the main driver behind the feature of automatically generating journal entries. Under this principle when an invoice is sent to a client a journal entry should be made recording we should get paid for that, ie. the client’s account receivable gets increased [5]. Likewise when we get a supplier invoice, an entry should record that we must pay that bill, ie. the supplier’s account payable gets increased.

You see now how the “Account Receivable” and “Account Payable” fields for the partner play their part in the automation of the accounting processes. This is deeply weaved into the account module’s source code. Meaning that there’s the assumption that partners have those properties we’re talking about. And that’s true because you have injected them and, if you configured everything as expected, they have their default values.

Notice the difference between the expectation of integration of business areas and how the integration happens in this case via a very specific kind of automation.

I’ll argue that the current state of this design is flawed. When standards change and/or are not applicable this kind of automation does more harm that it helps.

This is the reason the module that controls the “Anglo-Saxon accounting” [6] is very difficult to understand and the result artificial: they need an “interim” account to keep track the different stages. In the standard (for OpenERP) accounting the event to produce journal items in the debtor/creditor account is the creation of the invoice. In the anglo-saxon scheme the journal should be created at shipping time.

I argue that given another framework that clearly separates every actor and function will improve how this pattern could be implemented. I think that this framework must have:

  1. Signals and events.
  2. Actors like the accountant, and probably an automated agent for the accountant that could do the same the models do right now. But being responsive (ie. they respond to signals) they could be easily bypassed.

Of course there are more things needed. I’m thinking about those two plus the ones OpenERP already has.

I think that recognizing actors is the major improvement. Actors are abstractions about intelligence. If a person should be doing some kind of intelligent decision (like accountants), then you should encode (in your design) that decision as being taken by an actor.

Having artificial agents that could take over when the task is standard or programmable is also an option in this case. Anywhere in you design an actor does something, and agent could be replacing the human. The agents could be as dumb as the couple of rules we have now: create a journal entry each time an invoice goes to the valid state, and do it this way. But agents could be also provided of machine learning techniques and they could observe the how the human accountant proceeds when something happened. Of course this would require the human to proceed in case-by-case fashion and that’s not always true.

No matter if the machine learning is never done, I argue that designing with actors and agents will lead to better a implementation, easier to understand, maintain and evolve.

Notes

[1](1, 2)

This is no hypothetical at all. We’re actually deploying Accounting after having deployed Project Management and CRM.

This has come with many surprises but that’s what this post is about.
[2]The word addon in here is important. There are actually two OpenERP addons named “Accounting & Finance”: the account addon and the account_accountant. The second one is flagged as an application and, thus, it takes a more prominent place in the listing of available applications. Installing the application forces the installation of the account module anyway.
[3]That is why I still do my personal (home) accounting with GNU Cash.
[4]This merge has problems of its own, but that’s a matter for another post.
[5]Though an account either gets credited or debited, I will avoid that accounting-related terms cause it’s not needed for the argument in this post. If you need to know, start by knowing that receivables have a debit normal balance and go from there.
[6]I’m not quite sure if this “anglo-saxon accounting” refers to different basis of accounting.
]]> Wed, 02 Jul 2014 00:00:00 -0400 http://mvaled.github.io/blog/html/2014/06/05/odoo.html http://mvaled.github.io/blog/html/2014/06/05/odoo.html <![CDATA[Odoo… I mean OpenERP]]> Odoo… I mean OpenERP

This is old news: OpenERP has changed its name and now becomes Odoo. I don’t care much about the change in the name. It’s the processes I’m interested in.

The rename was synchronized with a move from Launchpad to Github. This, I much appreciate it. I’m one of those that once the Git virus got me, I’m almost unable to work comfortably with any other Version Control tool.

I this regard I have a couple of questions:

  • Will they use the “standard” Github’s pull-requests model for contributions? Probably they are longing for this kind of long-tail of contributors model to kick off now at Github. I haven’t found any “Why the move to Github” in the FAQ of the move and other sources (but again my connection is slow and I didn’t waste much time).

  • Will they require some kind of a “waive” of copyrights to accept contributions?

    The model of the Plone Foundation requires this. But this foundation is non-profit unlike Odoo.

    Does this align with the company’s view of the Open Source model? I don’t know.

So, my only strategy is:

  • Fork the Github repository
  • Send as many PRs as I’m able to fix/improve something and hope for the best.

Postdata

My long-planned “OpenERP corner” will simply be renamed accordingly.

This reminds me I have broken my promise to keep my old blog in-sync with this new one. Apologies. But it’s sooo comfortable to work with this blog-model that I barely looked back once. So I’ll announce it: I will not keep my old blog anymore. I will commit myself to improve the look of this one and that’s it.

]]> Thu, 05 Jun 2014 00:00:00 -0400 http://mvaled.github.io/blog/html/2014/05/27/tdd-is-dead-rally.html http://mvaled.github.io/blog/html/2014/05/27/tdd-is-dead-rally.html <![CDATA[TDD is dead rally]]> TDD is dead rally

These days there’s a kind of a rally about “TDD being dead”.

Most TDD fan supporters are true that you must have a way to proof your code works. Many computer scientists have done several attempts into the “proofs of correctness” for a program. But tests are not always proofs, some are [1].

I think the main argument is about when TDD is too much. Several Destroy all software screencasts show a TDD flow that I simply find it’s too much. But they are, indeed, screencasts; and they are probably intended to illustrate a point and not to be followed without deviations.

Nevertheless, I don’t like the “TDD is dead” mantra either. It gives space to a flame war I’m not willing to get into. I use tests and that’s it. Tests, at different, levels may express several concerns I need to keep stable:

  1. A public API, for instance, should not change just like that. At least not after you have made the commitment to keep it stable.

    You may even need to test for deprecation warnings being issued when you need to change an API.

  2. Collaborative in-progress debugging. This tests allow to express standing issues. Some regression tests fall into this category.

How do you do tests?

Update: Amplification and support to David’s arguments

The previous words were written just after skimming over several tweets. I had read David’s test-induced design damage but I had missed the previous post TDD is dead. Long live testing. So this update is my one cent to the issue, but I will not discuss about “slow collaborators” but about my design experience with our Python Query Language.

The test-first mantra assumes too much about how you are going to decompose your problem. Let’s start with a retrospective account of how this happened in xotl.ql.

I can detect four distinct stages:

  1. The beginnings. Slightly TDD, but the tests were not always written before code. Since xotl.ql is just a language we were not sure about what to test.

    The idea of having a query object that stands for the expression was not totally consolidated. The components of the query were not defined. This stage was heavily driven by our own writings, like this one.

    In this case the “literate” spirit dominated the design process, not the testing. Simply we didn’t have a full-stack: ie. the language and the translators so that a query could actually be executed.

  2. Consolidation of the design. In this stage several devices were invented to cope with implementation difficulties, i.e. the lack of clear boundaries between several sub-expressions in a single query.

    This was the hardest stage. Testing was employed to keep track of several design decisions. In this stage made appearance our “Particle Bubble” and this was complex enough to deserve a handful of tests. Although several tests did influence the design, they hardly drove it.

  3. Then the core was done and we turned to translation. This stage was mainly TDD cause we actually were testing a very TDD friendly layer: Given this state of the world, the following query should return these objects.

  4. Our current stage. Although in a long pause, we have decided to wipe out our entire design and do reverse engineering of python byte-code to extract build the query object.

    What is foreseeable is that again a kind of literate driven design is going to be king: The core structure of the language (the query AST) is what’s being designed so, what’s the point of asking if the result of calling a function with a given query expression is a particular AST if the AST is what’s needs to be validated not the function itself? At this point this is non-sense. At a later stage where the AST is stable enough those kind of tests would actually make sense: they will protect the query language against unintentional API-breaking changes.

Footnotes

[1]If I recall correctly, the book by R.L Graham, D.E. Knuth and O. Patashnik called “Concrete Mathematics” shows some proofs that allow “proof-by-instances”.
]]> Tue, 27 May 2014 00:00:00 -0400 http://mvaled.github.io/blog/html/2014/05/26/progress-in-the-move.html http://mvaled.github.io/blog/html/2014/05/26/progress-in-the-move.html <![CDATA[Progress moving to github.io]]> Progress moving to github.io

As you may see, I have made some progress. Basically, I have partially ported my past look and feel, so that if looks like the old blog. I still a work in progress and the “responsive” stuff is not entirely done.

Nevertheless, I think this is good to go and will start blogging here from now on.

Enjoy!

]]> Mon, 26 May 2014 00:00:00 -0400 http://mvaled.github.io/blog/html/2014/05/13/welcome.html http://mvaled.github.io/blog/html/2014/05/13/welcome.html <![CDATA[Welcome]]> Welcome

This post intends to be one to welcome my previous readers from Wordpress. I still have much to do to make this blog actually my blog:

  1. Polish a theme that resembles my current Wordpress look and feel. I like that central column, big fonts in big screens, unobtrusive reading and so on.
  2. Port my last posts to this new blog.

For the moment, though, this post is enough to avoid the default “Not found” message. For the moment, also, I’m using Tinkerer to create this blog.

]]> Tue, 13 May 2014 00:00:00 -0400