-->

..:: seize the day ::..

September 22, 2009

Flirting With Disaster - Linux for Emergency Operation Center in Indonesia

Filed under: my work

It’s a name of an interesting book "Flirting With Disaster" by Marc Grestein and Michael Ellsberg written on 2008. But this is not about that book and of course I will not flirting with the disaster, it is just to remain me not to play a game with my current project emoticon. Well, I live in Indonesia. Here we live with natural disaster emoticon. We’re sitting on top of ring-of-fire, just two weeks ago, in early September 2009, I experienced 7.2 SR earthquake shaking my office in 17 th fl. building.

I was asked by French Red Cross to help them make Emergency Operation Centers in Bali, Yogyakarta and Jambi in Indonesia. In short they ask me to install several Linux server for their project. I will not tell you the detail design of the system, but I will share some interesting experience that I think could be useful to implement in your installation. The most important thing, considering the nature of our country, I should think to design of the high availability working environment for the system to be built. Moreover the center should be functional in the event of the crisis, whether it is earthquake, flood, electrical failure etc. Well it is a hard job, but I think it is the time for Linux to show the capability for managing such situation in my country.

There are several function in the Operation Center:

  1. Web server
  2. SMS gateway
  3. Email and group ware server
  4. IP PABX server
  5. Database Server
  6. GIS Server
  7. Recorder and LDAP server 

I use several custom openSUSE 11.1 version that I build using SUSE Studio, thanks to SUSE Studio team that give the world the most wonderful re-mastering tool. We design the network on top of layer 2 HP switch with several VLAN, we also prepare the backup switch that already configure and when the main switch is broken the user can take off the switch and replace it with the backup.

We use Watchguard as router/firewall connected with two Internet connections that run on round robin high availability scheme. For the backup we prepare a Juniper that already configured that can replace the main router/firewall when it’s needed.

For the electricity backup we use a diesel generator that can supply enough power for the system, of course we use several UPS either for the server and for the workstation and other peripheral.

We design almost all the server in high availability with two servers running identical services in active-passive configuration. In implementing this, I consider the easiest implementation so the local support can follow our method and they can do minor troubleshooting by themselves. It is very important because I live far away from the sites. Of course I enabled remote connection using VPN and ssh but without local support it is not easy to do troubleshooting.

The implementation will be done in September - December 2009, so I will be very busy in the next 3 months. Nevertheless I plan to write my experience in several posting ahead.

Stay tuned and have a lot of fun.

November 4, 2008

Disaster Management Project 2007 (Part 4)

Filed under: my work

This post is the final chapter of Disaster Management Project 2007 (anyway there are some interesting story behind the project maybe I should also write a post about that). This post will talk about the implementation of project in Padang (capital city of West Sumatera Province - Indonesia), as well as the testing in Padang and Banda Aceh. Learn from the experience in Banda Aceh, I let the interior contractor do the job first. I don’t want to go there just to wait their work to finish. The provincial government of West Sumatera actually don’t have enough budget to make the room as complete as Banda Aceh. So I know from the beginning that there will be some handicap here and there, but it’s OK for me. We sometimes work like MacGayver so we’re not so worry :) . We also already sent all the goods to Padang before hands, and all the stuff already arrive one week before.

I went with Darma to Padang on November 30, 2007, two days earlier before Astrium and Infoterra team. In three days all PC and other peripherals are configure, the only problem is that (again, like in Banda Aceh) we should take an internet connection from another building. We put UTP cable around 70 m to get to the switch. For the server we need aproximately one week to configure, check the connection between Padang and Jakarta, and test the RiskFrame application. This time me and Darma are working fast. Meanwhile Hilarion and Karine from Astrium team go to Banda Aceh to install some modification for recorder server. They’re there 3 days and then join with me and Darma in Padang on December 2, 2007.

For you who don’t know where the Padang is, click here to get some info from wikipedia. Also here is the place in Padang where we put all the equipment from google earth.

Padang city from above crisis center on Padang

Padang is located in the west coast of Sumatera, one of active area of ring of fire. When we get there Padang just severe from 6SR earth quake 1 month before, and while we’re there we also get the opportunity to feel  the shake of earth quake. I was configure the server at 2nd floor at that time when a 5 SR earth quake shake the building, we are running out of the building and wait for about 10 minutes outside. We are lucky this earth quake is not long though :)

The first two days in Padang we are busy unpacking all the PC, server, rack and all the peripherals. We also setup the network, connect the cable to another building to get the internet access. Here are some picture back then,

Padang crisis center Padang crisis room

Padang coordination room Padang server room

Darma preparing UTP cable Padang spagheti incident

Me and the interior team The A team: me, Karine, Hilarion

During this project, it is in Padang where almost all the team member is present. From me and Darma from NSI; Hilarion Raobadia, Patrice Pessah, and Frederick Cathignol and Karine de Ponteves from EADS Astrium; Eric Joe and Marc Boulbess from Infoterra. From Civipol there is B. Neuville who will lead Site Acceptance Test (SAT). Eric Brossouloux from IGN FI will join in Banda Aceh on December 14 - 17, 2007 to follow the final SAT for Banda Aceh site.

There is one change in PABX/Asterisk server - recorder server configuration. In the design we will record all the conversation from Asterisk server to recorder server. We set the time 5 minutes before the recording file will be transfer from PABX server to the recorder. Means that we should wait for 5 minutes before we can playback the conversation from the PC in the center. Civipol ask us to add additional capability to directly access the recording conversation file after the telephone hang up, and they will include it on SAT. Antoher problem we have is the broken of one Digium card. I should ask our office in Jakarta to buy one and send it to Padang immediately. Meanwhile Darma facing no difficulties to configure all the PC, VHF radio, Printer, Satellite Telephone and Satelitte TV.

Padang switch Padang recorder rear

Padang recorder front Padang PABX front

Padang PABX rear Hilarion-Darma-me

SAT is running smoothly in Padang. It makes me happy. We’re running out of time because the delay from civil work. In Padang everything is doing in one take, from implementation, testing and site acceptance it takes only 13 days. I’m really proud of it. Here are some pictures of that period (well, you see everybody smile. It is sign of mission accomplished, isn’t it?) emoticon

Padang SAT 1 Padang SAT 2

Darma, Betrand Neuville, me Frederick the big guy, me small guy, smallest girl

On December 11, 2007 SAT on Padang is finished. Civipol can accept the site. It’s time to move to Banda Aceh. The French team go first to Banda Aceh, actually several hours before me. I should send some tools back to Jakarta, we don’t need that in Banda Aceh. The schedule in Banda Aceh is only for SAT so I don’t need that. Darma go back to Jakarta. He works so hard in the last two weeks, so I decided to let him take the vacation. Then I go to Banda Aceh. 

The flight is delay more than 2 hours. In the ticket it is schedule for 11.00, but actually I take off on 13.30. I should change the plane in Medan. At 17.50 December 11, 2007 I once again arrive in Banda Aceh.

On December 12, 2007, me, Eric Joe and Eric Brossouloux meet with person from Aceh GIS center. One of my friend work there under GTZ, and we need to gather the information whether we can use their map considering it might be more accurate than what RiskFrame has.

Bruno Maestracci from Civipol join with us in Banda Aceh on Friday night. He come with Paul Delavenne and Feni Rani. During that night finally every party accept our result and Civipol sign the Site Acceptance Test. This the end of this project, of course we should do maintenance it in one year.

OK let stop this story. I hope I can tell you other experience in another project.

‘Til then keep safe and stop global warming.
 

August 18, 2008

Disaster Management Project 2007 (Part 3)

Filed under: my work

We finished the work in Bakornas by the end of August 2007. It meant that we should continue to Banda Aceh (Capital of Nanggroe Aceh Darussalam Province) and Padang (Capital of West Sumatera Province). Banda Aceh call came first. It is the one of the outermost province in Indonesia, 2 hours and 50 minutes direct flight from Jakarta where I live, and 4 hours if we use the non direct flight. This province was hit by tsunami in 2004, at least 150,000 people died by that disaster (no one knows the exact number). In Banda Aceh city alone 60,000 people died. When tsunami happened almost a month we in Jakarta speechless every time we saw the news about it.  Everybody in Indonesia donate something for them. It was a great example of how citizen/people can joint together, doing good thing for humanity. It was very touching moment, many people went there as volunteer by their own money to help other fellow citizen, although at that time the situation in Aceh was not safe. There was a war there. Tsunami, in the positive side, cause every party to cease the war and finally signed a peace treaty. Now it is time to rebuild Aceh.

Indonesia is a country with a lot of natural disaster, living with disaster is the way we live. We are sitting on top of ring of fires. No other country in the world is like Indonesia. There are a lot of volcanoes here. This one a capture from the google earth.

indonesia ring of fires 

As for the Crisis Center location it is reside on the part of Nanggroe Aceh Darussalam Governor Office. The location is approximately on 5º34′15.68" N, 95º20′27.19" E with elevation 9m.

NAD crisis center location

For you who never visit Banda Aceh after the 2004 tsunami here are some pictures during my visit at the end of August and beginning of September 2007.

tsunami victim cemetary, near uleuleu one of miracle, uleuleu mosque stand still even only 30 m from shore line

kampong uleuleu vanish because of 2004 tsunami. Government decided to relocate the people. vessel power generator, 3 km from the shoreline.

Aceh Mosque stand still. The witness of Aceh history since 16th century Aceg Becak, common public transportation in Aceh

Ibnu team (the civil/interior contractor) already two weeks before us in Banda Aceh. And he assured me with his phone called that the room would be ready in two weeks. I can understand if there were missing thing here and there, but at least the room has carpet, electricity, and air conditioner in server room and I will go to Banda Aceh. So I booked non refundable airline ticket. This time I brought Darma, my fellow worker in office. He is young and a big guy, so it would be a big help to have him lifting heavy server upstairs in Banda Aceh he..he..he.. Anyway, since he is much younger than me, it would be a great experience for him under my coached as an "old networker guy" during this project.

Then I emailed Hilarion (EADS Astrium) said that I will go to Banda Aceh in August 28, 2007. He returned the email and said he would be in Jakarta by the time I arrived in Aceh, then he will ask the situation whether he should go or not go to Banda Aceh based on my report. I thought he was unsured that the room would be finished on August 28. I sent all the goodies a week before August 28, then I went there. When I arrived in Banda Aceh what I’m affraid is happened, the room did not ready. No carpet, no air conditioner, and the biggest problem server room was not ready. Below are some picture at that time.

Banda Aceh crisis center status on Aug 30, 2007 Banda Aceh crisis room status on Aug 30, 2007

Banda Aceh routine room status on Aug 30, 2007 our goodies for Banda Aceh

I quickly decided that I will stayed there not more than one week but Ibnu successed to hold me almost 2 weeks supervising his staff worked. When leaving Jakarta I was already on fire and ready to implement in this site because I knew everything better after implementation on DKI province and Bakornas. After saw the room status in Banda Aceh, I should change the plan. It would be giving some instruction to Ibnu and his member about electricity and grounding, position for tv satellite antenna, UHF radio antenna and the LAN cabling position. Also I should move all the goodies from the warehouse to crisis center location, considered that Ibnu team would be there to keep those goodies. I also reported to Hilarion that he did not need to come to Banda Aceh, better still in Jakarta and wait if the room can be finished in the next two week, which was actually did not.

One of the worst thing was that there are no internet connection in the room. The nearest switch was around 150 m away. So I decided to ask Aceh Governor IT staff to get an IP address and permission to put the cable from crisis center to that switch. Of course I should add one switch in the middle, simply for amplify the ethernet signal.

walk in to Aceh Governor office Darma reload the goodies

nearest switch to get the ip address for internet connection UHF antenna installation

Darma again on action nearest switch to get the ip address is 162 m away from crisis center

server rack installation tv satellite antena pylon

furniture for crisis center routine room satus as of Sept 8, 2007

After 2 weeks in Banda Aceh I decided to go back to Jakarta on September 9, 2007. In Jakarta I discussed the situation with Hillarion which would be went back to Toulouse on the next day. I plan to visit Banda Aceh again by the end of September 2007. By that time Bruno Maestracci and his team from Civipol also conducted a training. They conducted it without the server because we behind the schedule caused by the late of room preparation. Ibnu and his team already put the furniture, desk and some temporary chair for the room. Karine de Ponteves, my colleague as part of implementation team, went to Banda Aceh also with Bruno as trainer. She unpacked all the PC and several printer, but without LAN and server. It was my duty and Darma to prepare all the LAN cable and configuring the server.

Then  I went again to Banda Aceh with Darma at the end of September 2007. I remember it was on Ramadhan (fasting month for Moslem people) last year. At that time I changed our work-rhythmic, start at 11.00 AM up until midnight because we were fasting at the day. This time we finish LAN installation, server configuration, VPN and VOIP installation to Jakarta and testing all RiskFrame application which was already installed in the PC and server by Surjanto and Ericjoe in Jakarta before it was sent to Banda Aceh, also configuring radio and satellite TV. This time we were here for 2 weeks again. Below are some pictures of our worked back then.

NAD crisis center routine room aka Oct 2007 NAD crisis center crisis room October 2007

NAD crisis center coordination room Oct 2007 NAD crisis center crisis room Oct 2007

NAD servers and peripherals Oct 2007 NAD tv satellite antenna

NAD front rack NAD rear rack

By October 8th, 2007 our worked in Aceh is considered finished. I knew that I should be here again after Padang implementation for Site Acceptance Test with Civipol and EADS Astrium team. It was not decide yet at that time, but it will be on December 13-15, 2007. On October 9, 2007, me and Darma went back to Jakarta. It was an exciting experience worked in Banda Aceh for this project.

That’s it for Aceh. In the next post I will tell the story of implementation in Padang, West Sumatera province. 

 

 

August 12, 2008

Disaster Management Project 2007 (Part 2)

Filed under: my work

Back in Indonesia, I felt so excited. Only two days after arrived from Toulouse, I met with one of Deputy of Badan Koordinasi Nasional Penanggulangan Bencana (Bakornas PB). We discussed about the readiness of all site. And almost all sites except DKI were not ready yet. I started to worry because my work cannot be started unless the room are ready, and it will cause the delay of the project. Bruno Maestracci, the CIVIPOL person, really pushed hard to the Bakornas about room preparation. Sometimes I felt that the way he did the pressure can cause the friction to the Indonesian site. Anyway, without his pressure contribution, this project will never be finished.

Waiting the readiness of the room for DKI and Bakornas, I finished all necessary preparation including buying all the necessary equipment. Also I should went to Soekarno-Hatta Airport to ask the permit for all the goods coming from France. It was also the frustating task, because Indonesian official asked for import duty, which was I cannot fullfiled because it was a grant. So I ask Bakornas to handle this issues, as they are government bodies.

In the beginning of May 2007 all the goods are ready to distribute to DKI and Bakornas. So I send the goods to this two locations first. I worked on DKI first considering that the room is ready. In Bakornas the room is not ready yet. Below are some of the pictures at DKI office at that period. 

dki sphagetti incident  dki uhf antena

dki recorder & database server dki routine room

dki clean cable dki roof

dki antena dki staff busy

In DKI I felt the excitement of young staff. They were eager to use all the systems. There we are working very fast, me, Surjanto, Karine, Hilarion, Marc, Eric and anybody else worked so quickly to finish this site. By the end of May 2007, almost all the server, pc, peripheral already installed and configured in DKI.

Meanwhile in Bakornas, finishing of the room is running so late. I thought we were behind the schedule. In the end of May the room still not finish yet. Here are some of the pictures back then. These pictures was taken on May 15th and May 30th, 2007. According to Bruno from Civipol who provide the grant, he introduce the project since the end of 2005, so it should be aware by Bakornas staff that the room should be ready by the beginning of May 2007. I can Imagine how complicated the decision has to take on Indonesia Central Government body. I couldn’t imagine if they choose the same way to take the decision if the real disaster is coming, maybe it is too late to save the human life. 

bnpb crisis center at May 15, 2007 bnpb crisis center at May 15, 2007

bnpb routine room as of May 30, 2007 bnpb server room as of May 30, 2007

Behind the schedule, we should wait for another one month. In the beginning of June 2007, my colleague from Toulouse, Hilarion would be on Jakarta. Even I already told him that the room is not ready yet, he insisted to go to Jakarta. So in the mid of June 2007, even that the room was not ready in Bakornas we should install everything that we can install, and by doing this we hope the civil contractor will speed up the finishing of the room. Below are some pictures taken between 13 to 25 June 2007.

entrance of crisis center on bakornas bakornas coordination room as of 13 June 2007

bakornas routine room as of 13 June 2007 bakornas server room as of 13 June 2007

After discussed with Hilarion Raobadia and Patrice Pessah from EADS Astrium, we finally decided to do installation for the server and routine room, while waiting the civil contractor to finish all the other room. Basically we installed and configured all the necessary item, because in this project Bakornas system will be the heart of the system. From here we managed the voip system using asterisk for all the other sites, make ftp server for the "RiskFrame" application and also setting the vpn to the other 3 sites. We also need the installation of PC in Bakornas routine room so we can simulate some situation better. At that time the DKI site installation almost finished, only radio and satellite TV is not configure. By installing the server in Bakornas we also can check the communication between DKI and Bakornas, by doing this we can also simulate some problem that probably happen before we implement the system to Banda Aceh and Padang which is reside thousands km on the other island. 

Here are some pictures of us at the time we installed the Bakornas system at the end of June 2007.

bnpb asterisk server bnpb recorder & database server

hilarion in action on bnpb routine room bnp crisis room as of 23 June 2007

We needed more time in Bakornas because we need to wait the finishing of the room. Also we should simulate some condition that we found could influence the implementation in other site. After almost 2 months, thank God, finally the implementation in Bakornas is finished at the end of August 2007. The team from EADS Astrium and Infoterra also came from Toulouse to help the implementation ready on time (it is behind the schedule anyway). Here are some pictures of us at that the end of August 2007.

bnpb asterisk ready bnpb recorder server and network peripheral

bnpb rear rack1 bnpb rear rack2

We also set up several training on how to use the system and also the RiskFrame software. Here are some pictures during that activities in Bakornas. Some people are so eager to follow the training and some people are not. Anyway we told them that we try hard to fullfil the requirement of the crisis center, and now it depends on them on how to use and to implement it to help them in their work.

bnpb user training1 bnpb user training2

At the end of August 2007, our team consider that all the work in DKI and Bakornas were finished. I should move to Banda Aceh and then Padang to implement the crisis center for that two provinces. It was also very exciting experience. We should worked on unfinished environment, in Banda Aceh I can not patience so I should went there twice. Learn by Aceh experience I only came to Padang after all the civil works is done, which is actually unfinished.

I will write the experienced in the next post, till then keep safe and stop global warming. 

August 10, 2008

Disaster Management Project 2007 (Part 1)

Filed under: my work

I promised in my last column that I will write my worked in 2007 as flash back. For you who really impatience to read my story can directly go to EADS Astrium Press Release here. For the patience, here we start…….

Around mid-year 2006 I got the news from PASCO that we should help them for the upcoming project in Indonesia. The project will be from French Government and handle by EADS. My impression at that time is, it should be an important project. As far as I know EADS is a big company specialize in high-technology, they have rocket/launcher unit, satellite, Airbus, weapon and everything you can imagine.

Basically the project is a grant from French people for Indonesian people to help Indonesian Government to save human live in case of disaster. There are some areas they want to improve (hmm sound to good to be true for me ….), some training to increase the human (read public servant) capacity, some tools and equipment to improve the reporting and communication.

In December 2006 some EADS guys come to my office and have meeting with me and my director. To make it short, in February 2007 I should make survey in Jakarta, Banda Aceh and Padang with EADS staff and finally I should go to Toulouse France to EADS Astrium to do Factory Acceptance Test in 3 weeks on March 2007. The project it self should finish in September 2007.

I’m so excited. Working for the community is my passion, knowing that people use my work to improve the quality of human life making me very happy.

The schedule is very tight. In February 2007 I made survey to 4 location, 2 in Jakarta which is Badan Koordinasi Penanggulangan Bencana and Governor Office. Also 1 in Banda Aceh and 1 in Padang which is their Governor Office. I went to those location with Bruno Maestracci from Civipol who provide the grant, Hilarion Raobadia from EADS Astrium and Feni Rani who works for Bruno. This survey is to made sure that the infrastrucutre will be ready for the upcoming project which is included some computers and electronic equipment. The project is planning to start in Bakornas around Mei and follow by implementation in every regional center.

This is the picture of us on Padang and Banda Aceh

dr. Nessy, Hilarion, Bruno, me on Padang Airport      2 Sumbar Govt Office staff, Bruno, Feni, Nessy, me, Hilarion           

On March 2007, I went to EADS Astrium premises in Toulouse to do the factory acceptance test. I went there with my colleague Surjanto who will be concentrated on software. Meanwhile, I will concentrate on network and system. The network and system are consist of several server providing samba server, asterisk server, recorder server, ftp server, database server, as well as vpn and ip phone. The system is built on Linux machine with High Availabilty implementation using vrrpd. On software site there is a custom-made software which is called "RiskFrame" which will be provided spatial database for disaster and risk management.

Here are some picture of us on Toulouse:

me and surjanto, transit in Schiphol Astrium-Toulouse my work 1 my work 2 my work 3 Karine my friend from IGN FI me and hilarion la garone

After a month there, finally Civipol, the owner of the project can accept all the factory test. Now we are ready to implement the project. It is just the preliminary, the project itself will be in Indonesia. And the interesting, sometimes frustating moment just have to begin. It is all in my head when I was in plane from Europe to Jakarta. Can everything going well so I can finish the project on time?

I will continue the story in the next post.

Till then keep safe and stop global warming. 

February 18, 2008

Oracle [orakel]ar - kelar

Filed under: my work

Weekend ini gue, TN, WI, sama Candra dikerjain sama Oracle dan Microsoft emoticon

Gara-gara ada masalah listrik di client perusahaan gue, masalah sinkronisasi dari genset ke PLN, ada trouble di mesin cluster Windows 2000. Padahal mereka udah pakai UPS untuk ngatasin masalah-masalah kayak gini. Yang bikin panik adalah software yang running di mesin yang bermasalah ini adalah server untuk produksi, software ini mengatur "kanban" sistem, just in time process di sebuah pabrik. Waktu datang dan ngelihat truk-truk yang ngebawa bahan baku dan yang mau ngambil barang produksi pada ngantri, macet, gue bilang dalam hati "hell, we will stay in this weekend in the server room again"

Nothing wrong with the design, software dan server-server ini sudah berjalan hampir 6 tahun tanpa banyak komplain. Waktu kita buat sistem ini dengan Windows 2000 Cluster, yang "iklannya" adalah High Availability menurut si Bill, dan pakai Oracle database kita yakin sekali sistem ini akan tahan ujian dan cobaan, walaupun kita tahu kadang-kadang si Ora ini kurang suka juga sama si Mic. Hardwarenya sendiri walaupun high-end di tahun 2002, tapi masih handal untuk kondisi sekarang. Shared disknya pakai RAID ARRAY, yang belum obsolete. Sekali lagi nothing wrong for mission critical system.

Tadinya kita pikir ini masalah cluster, tapi waktu kita test read/write ke shared Itu service Ora gak mau jalan-jalan. Emangnya si Ora punya kaki apa mau jalan-jalan emoticon Gue sama TN coba-coba tulis di shared disknya, terus kita pindah-pindahin nodenya. No problem. Jadi kesimpulan kita clusternya OK. It’s around 9 PM Saturday night now. Giliran si Candra jajal kemampuan di si Ora, dengan sqlplus dan command line toolsnya yang gue kagak ngerti. Eh si Ora pundung gak mau konek. Gue lihat di Perf window itu proses read ke shared disk manteng di 100%. Gue dan Candra terus cari-cari error message nya si Ora ke temen kita mas Google. Intinya mas Google bilang itu si Ora gak mau baca karena dia menganggap udah ada session dan user yang ngebaca database, jadinya access deny.

Kesimpulan sementara ada masalah either di shared disk (bad sector, whatever) atau di proses instance-nya si Ora. OK kita sepakat untuk deinstall si Ora dan format ulang shared disk.

Deinstall si Ora sih gampang, sampai registrynya kita bersihin sekalian di node 1 dan 2. Terus kita matiin node2, karena kita mau format shared disknya. Eh si Cluster managementnya protes. Setelah pusing-pusing, mas Google ngasih tahu kalau mau format share disk di cluster windows, si cluster harus dibohongin. Caranya waktu pengecekan disk di cluster service harus dipanjangin. Kita set jadi 10 menit, baru si format mau kerja. he..he..he… gue bo’ongin lu. Gue tambahain chkdsk /f /r biar yakin gak ada bad sector di shared disk.

Sekarang tugas Candra untuk reinstall si Ora sama instancenya terus create ulang database dan restore. Jam udah 01.00 AM Minggu pagi. Setelah 2 jam berkutat dia nyerah. Selalu gagal berhenti di create database. Itu Disk read perfnya manteng di 100%. Gue amat-amatin lagi nih si node. Akhirnya gue ambil kesimpulan OK kita uninstall cluster servicenya dan kita create ulang. Supaya gak salah si TN gue minta capture semua Cluster admin ke bmp terus gue check satu-satu network card MAC Address dan IP Address. Jam 4.30 AM gue berhenti Subuhan dulu. Habis Subuhan gue lanjutin lagi dan karena ngecheck MAC Address satu-satu gue nemuin anomali di Hartbeatnya si cluster. Ternyata setting speednya NIC gak sama. Wah ini nih sumber masalahnya emoticon

 Candra on action

Jadi rupanya karena electrical instability, ini NIC ngereset akibatnya private ip untuk heartbeat gak sama speednya, yang satu 100Mbps/Full Duplex yang satu lagi Autosense, which is gak boleh. Gue inget waktu ngerjain cluster di Linux kalau heartbeatnya gue set di Autosense selalu aja gak jalan. Jadi gue set dua-duanya ke 100Mbps/Half-Duplex. Gue restart ulang kedua node, dan gue minta Candra untuk coba create database dan ngecopy backup datanya. Jam 07.30 AM Minggu pagi.

Jam 08.45 ini proses copy database ora kelar juga. Gue ngambil kesimpulan bahwa masalah yang terjadi adalah instabilitas pada Raid Array dan Fibre Channel mengakibatkan proses I/O yang ora kelar-kelar sehingga mengacaukan Cluster dan databas. Gue putuskan untuk ambil server cadangan di kantor untuk backup siapa tahu kita gak bisa ngejinakin yang pada pundung ini. Ternyata gak mudah juga ngatur orang di hari Minggu. Yang megang kunci kantor masih tidur, terus Bang Sahlan driver kitalagi nganterin TN pulang ke Depok, maklum istrinya lagi hamil muda jadi kita nggak bisa maksa dia untuk terus kerja. Eh gak tahunya Bang Sahlan ketiduran di Depok, jadinya itu server baru dateng ke Sunter jam 11.00 siang. Begitu saya periksa si server ini pakai harddisk SAS yang nggak ada drivernya di Windows 2000. Time is running out, gak sempat nyari-nyari di internet. Akhirnya sekitar jam 3 sore kita ambil lagi server lain di kantor yang harddisknya SATA. Kita install OS dan database. Terus ngedump data. Sementara itu gue terus juga ngebenerin si RAID ARRAY dan Cluster Server.

Singkat cerita jam 02.00 pagi hari Senin kita berhasil ngesetup server backup dengan database dan aplikasi yang siap dipakai untuk produksi hari Senin. Sementara itu gue perhatiin Cluster Server dan Raid Array yang gue benerin juga udah mulai berbaik hati sama gue, dia gak lagi ngabisin waktu untuk proses I/O yang gak jelas walaupun masih terasa cukup lama dibandingkan proses I/O di server backup. I give up dan saya usul untuk ngebawa orang prinsipal untuk nganalisa hardwarenya. 

January 8, 2008

My Work in 2007

Filed under: my work

I’m back.emoticon

It’s been a long year for me. I’m traveling so much this time. I don’t even have enough time to write. One month in Europe, one month in Banda Aceh, more than 2 weeks in Padang for Disaster Management Project, a Civipol France project that I work with my colleague from EADS Astrium, Infoterra dan IGN FI, for Bakornas Penanggulangan Bencana in Jakarta, Satkorlak DKI, Satkorlak Sumatera Barat and Satkorlak Nanggroe Aceh Darussalam. Besides that I also have several small project and other 1 big project for one telecommunication company in Medan and Makassar. These make the whole 2007 very busy.

I’ve met a lot of people, nice people and hope will continue as friend after this project finish. Let me take this oppurtiniy to thanks all of you start with French site:

- Bruno Maestracci, from Civipol, a Commander at France Legiuner, an emergency expert that works with "fires", thanks for the oppurtinity;
- "Opa" Hilarion Raobadia from EADS Astrium, a PhD from Paul Sabatier and "old-crack" satellite ground segment engineer who really passion about his works, thanks for everything, you’re my teacher;
- "Madame" Karine de Ruffi de Ponteves, he..he..he.. a high rank double de order, a software engineer from Paris, my co-workers who always proud about her heritage as east meet west inheritance, you’re the best;
- Feni Rani, an Indonesian who works for Civipol as Bruno secretary who always motivated;
- "the boss" Patrice Pessah from EADS Astrium thanks for believing me
- "the warmest and wisest guy" Eric Brossouloux regional director of IGN FI, please drop me an email.
- Jean Christian from EADS Astrium
- Eric Joe and Marc Boulbess from Infoterra
- the one and only "Pak" Frederick Cathignol from EADS Astrium (you’re my man) please wait me in Toulouse, I promise to follow your "Administration Manual"

On Indonesian site,
- my fellow NSI worker Wahyu, Darma, Surjanto, Uus, Alice, Roekhan please don’t forget that we should maintain these sites
- Ibnu Suharyo and the team, Kunaedi, Markus, Agus & Agus, actually you all make me nervous several time but thanks God it’s finished now.
- Pak Thabrani di Bakornas Jakarta, Pak Heru Joko dan staf di DKI, Pak Iskandar di Banda Aceh, Pak Suryadi dan Pak Ade di Padang.

and to all my family in Jakarta, Makassar and Munchen.

I will write my experience as flash back in several posting, till then keep safe and stop global warming.