A lot of time passed since our previous post.
It doesn`t mean that we haven`t found any serious problems. We rather haven`t found enough time to describe them here.
But on this week we meet something special.
The beginning was traditional - new arKItect release was delivered. This time - with incredible Flex LM licensing protection. At first all was normal - all test accounts were workable, only allowed users could log on and so on. On the second day we found that couple of internal KI accounts were disabled by Flex LM.
The first answer was: let's make an individual license for them.
Wrong answer.
Any complicated problem has a lot of simple wrong solutions.
They were simply missed from general list.
After rereading skype discussion for a last year we found that some accounts had been lost from initial allowed list from beginning. Not too complicated. Just update the initial allowed list and come to other problems.
After a couple of days we met this problem again!
Assumption way:
1. Looks like somebody replaces our allowed list by mistake - NO
2. May be internal program logic for supporting Flex LM is wrong - NO
Some developers look on each other as on Black boxes - it doesn`t simplify understanding.
4. Lets add more logs! At least it can help in future - OK. At least next step
Reread logs - looks like the problem is in FLex LM supporting program internally - no bad requests from outside in log.
Thinking... thinking... imagination of data flows in program... reading code... thinking...
Finally found (logically):
Due to the lack of protection from concurrent processes the Flex LM supporting program were breaking main license file in simultaneous reading/writing process.
Reproduced in real life on main server!
Reproduced on test server!!!!
Not reproduced after a fix!
It was impressively hard to find due to logical levels, and 3 minutes to fix.
It doesn`t mean that we haven`t found any serious problems. We rather haven`t found enough time to describe them here.
But on this week we meet something special.
The beginning was traditional - new arKItect release was delivered. This time - with incredible Flex LM licensing protection. At first all was normal - all test accounts were workable, only allowed users could log on and so on. On the second day we found that couple of internal KI accounts were disabled by Flex LM.
The first answer was: let's make an individual license for them.
Wrong answer.
Any complicated problem has a lot of simple wrong solutions.
They were simply missed from general list.
After rereading skype discussion for a last year we found that some accounts had been lost from initial allowed list from beginning. Not too complicated. Just update the initial allowed list and come to other problems.
After a couple of days we met this problem again!
Assumption way:
1. Looks like somebody replaces our allowed list by mistake - NO
2. May be internal program logic for supporting Flex LM is wrong - NO
3. May be External interfaces for this program are wrongly used - NO
no ways
Read short logs - nothing
Read long logs - nothing except exact place where allowed list was truncated
Some developers look on each other as on Black boxes - it doesn`t simplify understanding.
4. Lets add more logs! At least it can help in future - OK. At least next step
Reread logs - looks like the problem is in FLex LM supporting program internally - no bad requests from outside in log.
Thinking... thinking... imagination of data flows in program... reading code... thinking...
Finally found (logically):
Due to the lack of protection from concurrent processes the Flex LM supporting program were breaking main license file in simultaneous reading/writing process.
Reproduced in real life on main server!
Reproduced on test server!!!!
Not reproduced after a fix!
It was impressively hard to find due to logical levels, and 3 minutes to fix.
No comments:
Post a Comment