MASLAM – Vain Operation to Soothe the Conscience

I only had the pleasure of meeting the impressive Yoel Ben Porat in person twice.

He was a brilliant person, with a sharp, uncompromising view of reality and a great sense for inventing new idioms.

We came to interview Yoel for an historical review of an organizational unit that he founded. We were planning to create a glorified PowerPoint describing the many successes of this unit. Much to our surprise , when we asked Yoel Ben Porat to describe it  he just said “It’s Maslam”.

We never heard the word before, so we asked for an explanation.In Hebrew it translates to “מבצע סרק להרגעת המצפון” which is loosely translated as an Acronym for “a Vain Operation to Soothe the Conscience” (VOSC ?).

What he meant to say was – it’s a useless body, it never had any achievements, it will never have any achievements and  it does not get any real budget or attention.The only reason it exists is that the real solution to the problem is expensive , but we are too afraid to cancel the whole thing as one they this problem might actually take place.

This is quite a common phenomenon in Israeli institutes. For years, after terror attacks, the Israeli government would send the air force to bomb “destinations” in Lebanon to retaliate . The destinations were always empty, since the enemy was expecting it and the retaliation never worked.

The hourly  radio news had a constant the tag line, that almost became a joke, “Our forces have bombed destinations in Lebanon and returned home safely”.

Not surprisingly , civilian managers fall into the same trap quite often. Look around you and search for the MASLAMS you created -

• The special quality improvement project that lasted almost two weeks, but didn’t solve any root problems
• The security officer who is supposed to prevent security bugs, but does not get any budget
• The “We love the customer” signs posted around the office when most meetings are on “lets change the pricing to raise the profits”
• The “Lets improve the documentation project” by asking every developer to stay an extra hour a day to write an FAQ.
• The “New SMB product” that was created by disabling 80% of the  enterprise features, instead of creating a product that SMB really need

One thing that people hate the most is cognitive dissonance. When critical issues are at stake, put all the resources to work.

If it is  a MASLAM you are creating, you might as well cancel the whole project ASAP.

My Top 10 Rock Concerts

Swing your hips
Loose your head, and let it spin
Swing your hips
Loose you head, and let it spin
And we will look together
For the pan within

I was standing in line to buy tickets for the Scouts 90 years reunion (55 Shekel ). It was a long line as the same ticket office is also selling 2500 shekel  tickets for the Madonna concert on September 1st.

While I’ll probably miss the Madonna show, it reminded me of some of the concerts I enjoyed most.Here we go, not according to any specific order.

1. The Pogues in Brixton academy . London, UK, 2001

I flew to London especially for the Pogues reunion with my brother.

There were 4921 Drunk Irish and pseudo Irish in the very suitable surrounding of Brixton. Shane MacGowan looked like he is going to die any second, but still has not to this day.

People were singing before the show, during the show and all the way back on the Tube.

2. The Waterboys in Irving Plaza, New York, USA , 2001

It was a few weeks after 911 and the smoke was still rising over ground zero.Nick Cave has canceled his show due to the events,but Mike Scott kept his.

He played all the Classics with a great band and rock attitude. It seemed he is as nice and talented  in person as his persona is.

Waterboys in Israel Ticket - The one I didn't go to

The pan within is one the greatest Waterboys songs.Not as famous as “Fisherman blues” or “Whole of the Moon”, but it stands for everything great about the band. Watch the Violin solo.

3. Eran Zur – The Pilegesh Club , Rehovot, Israel, 1995

Eran Zur had only two “Carmela Gross Vagner” albums at this time, but the crowd was so excited that he played both albums two times ! Best chemistry I have ever seen. In the end he just said : “I have no more songs for you”.

Strangely, the the venue , a Rock club in Rehovot was a great fit for the show.

4. Red Hot Chilly Peppers – Baltimore Arena, Maryland , 2000

Coolest band in the world, at the time. Definitely the best live band I’ve seen.

No videos at that time yet . But this clip is pretty similair.

And another one , since the “Otherside” is the best song form the album.

5. R.E.M + Radiohead - Ramat Gan stadium, 1995, Israel

We got two for the price of one !

Radiohead were the cover band, just after “The bends” was rleased .What a great start.

Only 20,000 people came and some critics complained ,but seeing “Fall on Me” ,”So Central Rain”,”Country Feedback” and “Let me in” live in Israel can not match anything else.

And here is a very nice interview ( Dave Letterman ?)

6. Barry Saharof and Rea Mochiach 11Aleph – Jerusalem, June, 2006

It was four days before the wedding and we had to do something “different” before all the ceremonies.

An mazing album that conatins the (almost) only instrumental bit that I like.

7. The White Stripes – Primervare Sound Festival – Barcelona, 2003

I actually came to see Belle and Sebastian and Sonic Youth, both gave a nice show.

However, the White Stripes’ energy , at 2AM in the morning came out fresh, massive and made everyone else seem “old”.

I had to fight to get a ride back to my hotel, to make it on time to my 6AM flight back to Israel. No time to shower at all ….

8. Nick Cave -Haifa Blues Festival , 1995

There is no such thing as a bad Nick Cave concert.This specific one was set in Haifa’s port and the rugged atmosphere felt right.

P.J Harvey was a great intro and it seems Mr Cave felt at home in the middle of a blues festival.

I kept the T-Shirt till 2009.

[youtube-http://www.youtube.com/watch?v=TAf4sIWUMK4]

9. The Pixies &  RHCP – Phoenix Park on Saturday June 12th,Ireland, 2004

The pixies are not a great live band, and RHCP were not as good as in Maryland. Still, it’s the Pixies , Live in 2004 !

Took us about 3 hours walk from Dublin center to get to this huge park ( with a stop in the Guinness brewery ) and 3 hours to get out of it.

It it is 7 times the size of Central park …

Phoenix Park , Dublin

10. Morissey in Tel Aviv, Israel, 2008

Yes, he looks like Elvis now. But that’s a sort of compliment.

And “Irish Blood,English Heart” seems like a good sign-of for the hidden Irish theme of this post.

Lovely Google Calender Error Message

Google Calnder Error Messages

This has to be the most language rich error message I have seen.

A Mexican, a Sicilian and a Greek Walk Into a Restaurant

This post starts like a joke, but it’s  actually a true story.

Funny Cat and Dog Eating

Next to our Virtual Lab Automation start-up (notice the subtle SEO promotion ) there is a corner restaurant spot.

In the last two years it occupied a Greek restaurant, A Mexican eatery and A Sicilian place. A new venue is being built as we speak.

Located just across the street  is one of Tel Aviv’s  most successful and long running restaurants -  The Coffee-Bar.

Why does one restaurant work so well and why do others fail one after the other ?

It can’t be just the location, since it is crime infested in both sides of the street.

While I don’t pretend to know a lot about running such establishments , I was able, together with my esteemed colleagues, to detect some generic management failures that can shed light on this huge gap.

The Sicilian

Design - The Sicilian restaurant had pretty good food. Their Sweet Potato Ravioli was excellent and so was the antipasti. However, the place had no warm feeling of a restaurant. It actually didn’t have any feeling at all. It had too few tables, lost in a big space. The main design element was a red children scooter. From the street it was next to impossible to get a feel of the restaurant.

Marketing – The second problem was that it always had too few people inside. So even if someone walked in he would generally walk right out , because it is a bad sign to when a restaurant is .We were so worried about them shutting down we offered to help them out, but nobody took us on our offer .

If they would have attracted enough people for the first time, many would have come back, since the food was good and service was reasonable.

Pricing -the business lunch was a great deal. For the price of the main course you would get drinks, antipasti, focaccia, green salad and a first course in a bundle.

But at the evening their prices nearly doubled , positioning them with the top gourmet restaurants in Tel Aviv, which didn’t really align with the design and ambiance.

The Greek

Design - they were able to take the depressing design of the Sicilian and make it even worse. When we first entered the restaurant we thought they have not finished the remodeling.  The floor was fully covered with wet paint drops. the kind you have when you decide to paint the walls on your own and then spend lots of time removing it with turpentine.

Service - in most of the times we dropped by, the cook was speaking in his cell phone behind the bar and the owner was smoking next to him. The waitress that  had a lot of good will, but would bring the courses in the wrong order. One time she cleared the dirty dishes from another table , came to us to take our order and placed the filthy plates just on our clean tidy map.

Positioning - on one of our visits, when it just opened up,  the owner asked us for feedback. Ori , being an open and honest person volunteered to say we enjoyed the food, but the meat was pretty dry. The owner explained that in Greece the meat is cooked with a nice dairy yogurt that keeps it from drying. However, since her restaurant is Kosher she can’t do that. Well, if you can’t serve good Kosher Greek food, why would should we eat it ?

Furthermore, there are two good, reasonably priced Kosher restaurants 10 meters away (Libyan and Israeli ) , so the category is quite crowded. Greek food is not amazing , but if you take away its uniqueness , there is not a lot left to desire for.

Oh, and they served free Ouzo for lunch. We program in Python for god sake. There is no compiler to save the day!

Drinks for Lunch ?

The Coffee Bar

Consistency – they always serve great food. The food quality is amazingly consistent, and in the rare cases they have a mistake , they would make up for it and apologize.

Service - the waiters are highly educated students, with a good combination of intelligence, professionalism  and a slightly cheeky attitude. Despite the fact that most work there for many years, we can see them being briefed twice a day for over 30 minutes.

Pricing - business lunch is a bit more than expensive than the  other options, but given the great quality the ROI is quite good. The dinner menu is 30% higher, which is reasonable. Since the place is always packed they don’t really have a reason for a change.

Lesson of the day

It is not easy to be the best in your business, but it seems some guidelines are always true. Day to day professionalism, consistent service, listening to customer feedback and the right positioning are the  key. It is easy to spot quality when you see it, but not always easy to copy.

I’ll update on the new restaurant status once they the remodeling is over …

Horses and Monkeys

We had some internal debate if the following pictures would work well as marketing campaign. I came up with the concept and our graphic artist created it.

I was giving them out as postcards next to the booth in VMWARE Partner exchange.

Some initial conclusions:
1. Animal pictures do attract attention.
2. Monkeys are better than horses.
3. People like humor.

4. If you hand out something to people, they stop and look at it.

5. If people start thinking you have time get their attention.

free the sales monkey vmware partner exchange

j

unleash sales horses vmware partner exchange 2009

vcloud booth IT Structures VMWARE partner exchange

Code Review Checklist

Here is a plea
From my heart to you
Nobody knows me
As well as you do

Depeche Mode, Ramat Gan Stadium , May 2009, Assuming no new war in the next two months

You don’t have to do code reviews, but if you do, do them the right way.

Code review  is a managerial process, IMO, and should be done by managers, not peers. The reviewer does not inspect just the code, but also the development process, guidelines and state. For example. If a peer detects a problem requiring a change in the code, a fix will happen for the specific problem. If the manager detects a difficulty in the code, he can make sure the cause is fixed. Peer review is an interesting tool and has its own benefits, but the two should not be confused.

The great benefit for the reviewing manager is education. He learns about the real problems his engineers are facing, the real status of code, documentation and people skills.It also allows him to stay connected with the source. It is  a great trade-off point between easy-to-shout-global-always-true declarations ( “No code with printf, ever”) to the reality of day to day (”printf is ugly, but it works”).Of course, he gets to learn more about core issues  in the code worth fixing. The same issues that he never prioritized.

There are some nice guidelines here http://www.macadamian.com/codereview.htm and below are some bullets you might want to use in your own process.

1.       Was the change tested to work in real environment?

2.       How do you KNOW if does what it is supposed to do?

3.       Are there potential similar bugs in the code? Were they fixed?

4.       Is the change\feature documented in the code?

5.       Are there unit tests to check the new feature?

6.       Are there any compilation warnings? Why?

7.       Is there an open ticket for the change?

8.       Has the change been checked for security issues?

9.       Did you  check memory allocations and releases?

10.   Is the change accompanied with right debug messages and logs?

11.   Functions have clear descriptions, return values, input and output

12.   Standard type definitions are used

13.   No arbitrary constants are used in the code

14.   What are the risks involved in the change? How are they minimized?

15.   Does the reviewer fully understand what the code does?

16.   Are state machines documented as such (clear states and transitions)?

17.   How will the change affect performance? Did you run a profiler?

18.   Was the design document updated to reflect the change?

19.   Textual messages should be check for spelling and clarity

20.   How are errors, exceptions and lack of resources dealt with?

21.   Can the new functions be useful to other team members? Did you tell them about it ?

Cat Doing Code Review

The Joy of Instant Gratification

My poor employees know that I want everything to be ready right now.It’s a bit childish, but I can’t fight it.

I hate waiting 3:30 minutes for Windows to boot.It’s like watching paint dry.You really don’t have to watch all of the video. But there is a funny cat in the beginning and in the end.

I agonize when I have to work with SalesForce EMEA slow servers.

I want every IT Structures customer to get a 300GB environment, 8GB memory with 15 servers and five VLANS in running in 10 seconds.I spend my weekends reading on RDP registry hacks in order to improve the user experience into instantaneous feedback.

As the great Freddy Mercury said it

I want it all and I want it now

It probably didn’t work that well for him in the end.

It seems that most vendors don’t get the need for instant reward.

It is well known that humans react well to short, timely  feedback  but I just waited 10 seconds watching the annoying hourglass icon  because I tried to install Babylon while writing  this blog entry. My hard disk crashed, so I’m reinstalling everything from scratch….

This brings me to the great next clip by Samsung. This is the hard soft drive that I want !

It fires up all of office applications in 0.5 a second  and would probably never crash on me since there are no moving parts.

And in the same notion , IE8 is finally out so the Microsoft  browser actually makes 6 HTTP concurrent requests instead of two.

I once hoped it will speed up the Web by 20% . Lets see if it does. In the mean time, maybe I should learn from the French to enjoy the slowness in life.

People reading in Luxembourg gardens ,Paris, leisure in the winter

musicians in Paris winter, Luxembourg gardens

boats and ducks in Paris winter, Luxembourg gardens

A Case Study in Restore Nightmares

I’m learning to fly but I aint got wings
Coming down is the hardest thing

Well the good old days may not return
And the rocks might melt, and the sea may burn

Tom Petty

Read the first part of the story first.

Few hours have passed and the Sasha felt more secure about starting the recovery of ClearCase source files.  The initial restore was successful (300GB and 50Million files) and everyone was happy. Everyone but yours truly.

Me: How do you know the recovery was successful ?

Sasha: The recovery software says so.

Me: Well, we know what that’s worth. How do I know all the files are OK ?

Sasha : We will run FSCK on all file systems and see what happens.

Me: That’s a good start, but it still does not tell me which files we lost, what data is corrupted , and that no changes happened. And by the way, how long it will take to run FSCK ?

Sasha: These are excellent questions, I don’t have any idea. Let’s try and see what happens.

Backup and Restore Case Study

Running FSCK took a few more hours, but it seemed Ok. We tried to bring ClearCase up but it was not willing to. Surprisingly the restore process didn’t restore all the file permissions and soft links. These had to be added manually.

I was still unsure we didn’t lose all the information. With 300 modules, 20Million files and thousands of branches it is really hard  to know nothing was lost. It could take months before someone tries to build an old package and finds out it is missing. With databases there are multiple Integrity checks in place.With a Version Control system such as ClearCase it is not so simple.

When we called ClearCase guys (also IBM) it turned out there is a secret script that validates the internal consistency of the setup. We decided to run the script and at the same time build all the main products on the compile servers. If all the compilation results come out identical to what we have we can be quite confident we have at least the latest source available.

What we soon discovered is that we can’t do both at the same time. The hidden script was locking the files. We had to run the build after the script. Of course nobody in IBM could estimate how long the script would run .I sent all the developers home at this stage.

A Couple of hours later everything seemed to be in order. I went back home and prepared for the holy day. Yom Kipur is the day in the year when Jews are supposed to ask for forgiveness for the evil they did to their fellow men.It is also a day for reflection I felt this was very appropriate opportunity.

Lessons Learned:

• A backup without frequent restore exercises is like a Pizza with no cheese
  • Just like High Availability never works in Passive Active mode
• Everything starts from the requirements. IT is not different than R&D
  • Restore time is one example
• If you can’t prove that the restore works you have a serious problem
  • This is a hard one. Think – what will make you sleep well at night.
• When crisis happens, it is very nice to have a process in place.
  • Start from the phone list of key people
• Trying to save money might cost you lots of money
  • There is usually a reason for high-end products
• Trust no one
  • If they say they have backup. Ask them for the DRP plans
  • If they claim to have backup ask them to restore something

A Case Study in Backup Nightmares

It was a month before Yom Kippur, the holiest day of the year. The IBM Central Storage server sent out a distress email to the Sasha, our sys-admin – “My backup battery needs to be replaced, three years have gone by and I’m going to depart this life in a month”.

“Time travels quickly when you’re having fun”, thought Sasha. He was not very worried as this was just the backup battery in case both the main power and UPS fail. For the geek readers – the battery allows to write the cached files from memory to the disk.

Our lovely sys-admin called up his IBM reseller, but unfortunately they were out of stock for this model and promised to call back. Three weeks later, the battery sent a reminder:” I’m moving to a better world in a week, please replace me!”

Sasha realized the reseller never called him back, which was not a huge surprise. He called them again. The reseller still didn’t have the right battery, but had a compatible model authorized by Big Blue. A special replacement ceremony was scheduled just two days before the holidays.

Replacing the battery went well. Strangely, it required turning the server off, but the beast came up again nicely. For exactly one minute. Then it completely crashed !

Then they called me, poor director of development, also in charge of development infrastructure.

Me: What’s Up ?

Sasha : The main storage Shark is down.

What are the implications ?

Sasha: We don’t know. We think everything is down.  SAP, ClearCase, ClearQuest , personal home directories.

Me: Where is the  IT Director, Mr Wolf ?

Sasha : Two weeks’ vacation in Italy.

Me: Where is the backup admin?

He left the company one month ago, no replacement yet.

Me: I’m coming over.

When I came over the place was in mayhem. The CIO has arrived , but he didn’t have a technical clue of what’s going on.

Naturally the first thing Sasha tried to do is to restore the storage from the backup tapes. He quickly found out that the index file for the tapes is stored on (pause here, embrace yourself) the central server itself.  Sasha blessed the idiot that decided to put it there, just because it was the fastest storage available.

“Never mind”, he thought, “I will restore the files manually”. Little did he know that the Tivoli restore software has a bug. If   the tapes are delivered in the wrong order the daemon crashes and all the restore processes have to be started from scratch.

I came over to poor Sasha.

Me: When will ClearCase be up and running ?

Sasha : I don’t know. We are currently restoring just SAP, because we are afraid if we make a mistake in one of them it will kill all the others.

Me: OK SAP is more important, but when are you going to finish restoring it ?

Sasha : I have no idea, in the current rate it would talk around 22 hours.

Me : What do you mean “I don’t know” ? Don’t you have a DRP in place ? Isn’t the recovery time the key to building the DRP setup ?

Sasha : No. We can only measure the current rate and guess the future.

Me : Why is the rate so slow? , the whole company is stuck and Yom Kippur is approaching.

Sasha laughed sadly.”We wanted to buy a backup system with fives tapes, so restoring can be much quicker. But our CEO thought that this is much too expensive, because in the normal times backup only one tape is busy. He hates paying for idle equipment”.

To be continued …..

The Questions Reporters Never Ask (But Should)

In the technical field bloggers can be more professional and accurate then analysts\reporters  because they actually use the products they talk abort in the field.

I’ll demonstrate with the top questions you never hear analysts ask, but they should be asking, IMO.Lets assume a company just a announced a new version of their cool, innovative database. Here is an imaginary dialog.

Question:  How many memory leaks does your new Database have ?

Answer : We don’t really know because we didn’t find all of them, we have a watchdog that kills the daemon every 6 hours. Marketing invented a great name for it – “Automatic recovery in resource constraint environments”.

Question: If the new features are so good, why are they turned off by default ?

Answer: we thought that the real customers would not appreciate the 90% decrease in performance, but still wanted to get the “innovative vendor” award.

Question: How many bugs have you decided not to fix in this version ?

Answer : Only 500. We decided that if the customers haven’t complain about the 5000 bugs in previous versions, we should be fine this time as well.

Question: Why are you bundling the automatic scheme UI with the basic package now ?

Answer: no one would buy the stupid thing as standalone product. This way we might make some bucks out of it.

Question: Can I get the product to play with for 30 days ?

Answer : God forbid. Next thing you’ll want to talk to real customers and not just the two we found for you.