Computer Related > Large Project Data Transfer Computing Issues
Thread Author: zippy Replies: 30

 Large Project Data Transfer - zippy
Monday morning and its new project at work time.

Just come out of a conference call and I've been asked to help out on a major project. Not expected to do too much, just some sense checking.

We purchase one of our core IT systems for my specialist division from an industry specialist provider and currently use version 1 from about 1995. Basically its an off the shelf banking package.

We are upgrading to version 10 from 2020 - well sort of. The budget is I understand £1m+ and they are at the final stages leading up to making the software live.

I am really surprised that the software supplier expects all customer accounts to be transferred manually - static and live data - to be typed in - thats thousands of transactions for some customers.

Surely this is not normal and a program would normally be provided to match old fields with new and transfer the data over?

Before I say something I just would like to know if anyone else has had to transfer data from one system to another with the same supplier and what the process was.

(Confirmed by the in-house IT lead and operations manager that manually transferring records by retyping them was required. Haven't been able to speak to the IT supplier yet and wont until I get a better understanding.)
 Large Project Data Transfer - Manatee
I'm not an IT specialist but that sounds unfeasible.

I've been involved with several large data migrations as no doubt you have although usually to a different supplier which if anything would be more challenging. All were mapped and bulk converted. At least any errors are likely to be so gross as to be noticeable. The first one I was involved in was in 1987 when the conversion managed to cancel the direct debits for all the storecard accounts for a particular retailer, which put almost the entire file into arrears after the 1st of the following month.

Somebody must have the wrong end of the stick. Could be me of course. You'd have thousands of errors doing it manually, not all of which are likely to be detectable.
 Large Project Data Transfer - tyrednemotional
....been there, done that, many times over the years.

Probably the first biggy was the conversion of all our systems (which were data hungry) from ICL1900 to ICL2900 platform, which you would have thought should have been easy, but they have different architectures and file systems.

All the files were "transformed" using specialist software from a San Francisco based outfit, which ran on IBM 360 machines.

The most complex part was defining "mapping" rules to convert from old to new, but once done and tested, the process was entirely repeatable and consistent.

I can't believe similar facilities don't still exist, after all that was over 40 years ago.
 Large Project Data Transfer - smokie
I'd have that high up on the project risk register for many obvious reasons, not least the time it might take but most critical the scope for error.

It does sound unbelievable for this day and age, and I'd be questioning it. I wrote software to transfer data between formats on many occasions when I worked for a manufacturer. It wasn't really very clever software as it was usually something we were doing without a budget and proper time allowance. But it was always do-able.

I can remember in some migration projects there would be a bulk load of historic data up to a particular (recent) cut-off point, and anything after that had to be treated differently, but I can't ever remember that that involving re-keying it.
 Large Project Data Transfer - Falkirk Bairn
Re-typing = madness - error prone
What is data format from 1985?
What is the database for the new system?
 Large Project Data Transfer - Bromptonaut
>> Re-typing = madness - error prone

That's reminded me....

A couple of years either side of 1990 I was managing a cashier section in a small government agency. Two different legacy systems. One running in some ancient Kienzle machines, apart from giving me grey hairs over PAYE on a small pension payroll it ran OK.

The other used a legal accounting package called SOLACS which as was the way then was supplied with the hardware. Burroughs kit. Designed to be run without specialist IT skills but with manufacturer training and support by. By the time I inherited it the trained staff had moved on and support, following various mergers, was non existent. Eventually we handed the hardware over to the IT 'professionals' in their air conditioned den on the 4th floor. My staff ran the processes and keyed in payment/receipt data from handwritten forms authorised by responsible staff.

After one of its many breakdowns the 'professionals' restored the data from the back up tapes but used the wrong tape for one half of the data. Two or three working days later, including a Tuesday when we ran a weekly regular payments run people from the 'client' department were beating a path to my door reporting inconsistent account balances and what were reported as chain errors.

We had to re-key about a week's worth of work. The automated regular payments could be re-run but because of the balance differences and fact that it wouldn't make payments that resulted in a negative ledger balance on any account the run totals differed by several thousand. There were some differences in the keyed runs too but they were fairly easily reconciled.

I spent an entire weekend going through dot matrix printed reports identifying differences. A lot of the errors involved £150.00 as that was the maximum amount for cashable warrants. Eventually got it down to a manageable figure by Sunday night and finally found a couple of transpositions.

That was less than a week's work re input in a very small operation.

 Large Project Data Transfer - Zero
>> I'd have that high up on the project risk register for many obvious reasons, not
>> least the time it might take but most critical the scope for error.

High on the risk register? yee gods, is there a project manager? how did this ever get past procurement stage?

No provider would sell anything if the requirement was manual transfer.

BUT - I suspect the clue is in the V1 to V10

You need to look at the procurement process.
 Large Project Data Transfer - tyrednemotional
>>
>> BUT - I suspect the clue is in the V1 to V10
>>

...the problem is converting groats to pounds....
 Large Project Data Transfer - Zero
Yer, From V1 to 10 we could be transferring apples to pears, lbs and oz to kilos, £sd to Euros.
 Large Project Data Transfer - smokie
It's only bits and bytes at the end of the day. The problems could arise I suppose with maintaining relationships between lumps of data but that's something that a proficient programmer should be able to manage (well I used to, and I never considered myself that good!!)
 Large Project Data Transfer - Manatee
Heavily customised and badly documented? Even so...
Last edited by: Manatee on Mon 1 Jun 20 at 15:08
 Large Project Data Transfer - Manatee
>>they are at the final stages leading up to making the software live.

That was when I was asked to join the first one having just joined the business. Panic had already started. There was a project meeting at 6pm every day to review progress on known problems and unveil the new ones. Of course they were breeding. I have never seen so much stress in one very smoke-filled room.

There was no capacity for parallel running. There was a pilot conversion, no way back but at least it was only maybe 10% of the file and one client that went wrong.

Happy days, but only in hindsight.
 Large Project Data Transfer - zippy
Hmm. I made the mistake of finding a huge omission the week before a huge project was about to go live with a subcontractor a few years back which meant we had to dump the subcontractor as they couldn’t do what we actually needed from the contract.

I wasn’t involved in the project, just asked to review it and the contracts before it went live.

This must be karma’s revenge!
 Large Project Data Transfer - No FM2R
>>they are at the final stages leading up to making the software live.

And how are they doing the testing then, if they're in the final stages.

No testing and manual data transfer is *so* bad that without meaning any offence I wonder if you have misunderstood?

Certainly you need to go and discuss this, offline and privately at first, with whoever has asked you to look at this. Don't go directly to the supplier,or anybody else for that matter, go first and quietly to whoever asked you to review it.

That will avoid the chance of you making a fool out of either yourself or the other guy.
 Large Project Data Transfer - Zero
I need only say one thing

TSB


Check it out.
 Large Project Data Transfer - zippy
>>testing

It’s effectively off the shelf and other companies are using it so I doubt there was any real testing and no one has bothered talking to rivals to see what problems they had!
 Large Project Data Transfer - zippy
>> talk to...

Have already made a few subtle calls and it looks like this is being supplier led.

Person running the project in house has no IT project experience and has been told by the supplier that the data needs be transferred manually.

Will make more discreet calls tomorrow. Supplier is pushing this through because they think banks are going to pull funding in light of CV19!

I suspect the supplier has no one around that knows how V1 really works!
 Large Project Data Transfer - Falkirk Bairn
Reminds me of an in-house project SQA in roughly 2000.
SQA are the Scottish Exam Board - Schools & FE

They attempted to write new records software - moving from 20 year old Cobol to a database. Budget was some £4m - dug themselves a very deep hole - exams were marked and collated on a spreadsheet so results could go out.

One Glasgow College Student got an A pass in Italian - the nearest he was to Italian was a pizza.

Digging them out of the hole they dug cost the Scottish Government £36million. When you are on fire everything bought in gets a lot more expensive. They bought 2 quite big unix boxes - why 2?

The hardware man at the SQA connected the new Unix server to a 3 phase supply & fried it.
 Large Project Data Transfer - Kevin
>We purchase one of our core IT systems for my specialist division from an industry specialist
>provider and currently use version 1 from about 1995. Basically its an off the shelf banking
>package.

Let me get this straight.

You are running one of your "core" IT systems on 25 year old software? What's it running on - Win NT/SVR3 ? You now want to update it to the latest version and complain that the supplier doesn't provide a tool to migrate data through 9 different versions in a single step?

Suppliers will usually provide a tool if migration is required between one version and the next ie V9 to V10 but expecting them to also cater for 25yo data formats is ridiculous.
Last edited by: Kevin on Mon 1 Jun 20 at 22:57
 Large Project Data Transfer - zippy
>> Let me get this straight.
>>
>> You are running one of your "core" IT systems on 25 year old software? What's
>> it running on - Win NT/SVR3 ? You now want to update it to the
>> latest version and complain that the supplier doesn't provide a tool to migrate data through
>> 9 different versions in a single step?
>>
>> Suppliers will usually provide a tool if migration is required between one version and the
>> next ie V9 to V10 but expecting them to also cater for 25yo data formats
>> is ridiculous.
>>
>>

To be fair, it has nothing to do with me why the company is still running the old software. I guess that it did the job it needed to do, with no real need for an upgrade. I know other companies are still running v1 and v2.

It is running on the supplier's servers. That's what's so annoying about it. The supplier has always promised an easy upgrade when it was needed and in fact I was at the original sales pitch when that was promised. I voted against the upgrade because I wanted to see what alternative suppliers offered but the management team decided to stay with the current supplier.

The software has been maintained over the years.

I haven't been involved in the project and have been called in at the last minute because of the sudden unanticipated workload.
 Large Project Data Transfer - smokie
In my last job (this was about 6 years ago) I worked on a major bank's desktop refresh which was using a piece of 3rd party software on it's front-office PCs which depended on (I think) EGA screen standards, with corresponding dependencies in the back end software.

The machines we were migrating to didn't support EGA and it caused no end of problems. The 3rd party had no-one with the detailed knowledge of how the product had been built, so what should have been a fairly cheap and cheerful upgrade took months and a bucket load of money. We did look at installing additional graphics cards but the bank decided against that.
 Large Project Data Transfer - No FM2R
You probably can't see, but I'm crying.

No testing?

How do you know it works? Perhaps you're using an obscure piece of functionality that is rarely used. Perhaps you are using modules or transactions in conjunction with each other which are no longer supported. Perhaps any reports you get will be different. Perhaps any data downloads for subsequent use by Excel or VB will be in a different format. Perhaps not every data field is in the same format that it was before. Perhaps there are additional data fields. Perhaps the functionality has been 'improved' (that means changed). Perhaps not all the same fields are on the same D/E screens. Perhaps things have to be entered differently..... etc. etc. etc.

Manual D/Entry?

How will mistakes be corrected? Have any format requirements changed? Are the same number and type of characters in each field? What happens with D/E errors? Will there be reconciliation reports? Is every data field required by the new version exactly mirrored by a field in the older version? What about typos? How long will it take? By how many people? Can the system be used in this time? Presumably both systems will need to be locked out during the d/e exercise to avoid transaction changing data? What about interfaces to other systems? What about transactions which would normal occur in this time? Will the required parallel running be possible?

et friggin' cetera.

It's one of your core systems and you have no decent lead and the supplier is telling you what will or will not be done.

What a total nightmare. Even if everything goes perfectly it will still screw your operations for a few days.

Who asked you to do this sense check? Tell them, immediately, in writing, everything you're not happy with. Because you've been asked to "sense check", thus any continuing nonsense is happening on your watch.

You don't necessarily need migration programs, they're frequently used when there are may interfaces or custom functionality involved, so it depends on your particular environment.

But you certainly do need a data migration strategy. Whether or not it will be automated.

And you abso-bleedin-lutely need testing. Of everything.

And unless your supplier is guaranteeing to cover the cost of any disruption to your business, then I suggest that someone stops letting them make the decisions.
 Large Project Data Transfer - zippy
>> You probably can't see, but I'm crying.

Not as much as I am. It's been a hell of a morning.

>>
>> No testing?
>>

I know. I went through similar scenarios with the in house team yesterday. If madness. I found out one of my department's relied on reports will no longer be available - though a replacement can be written for a fee!

>> Manual D/Entry?
>>

Requirement is to key twice. Differences then reported - we are not in the '60s FFS.

A quick back of the fag packet calculation suggests over 1m entries of static data (names, address line 1, address line 2 etc). There are going to be no b/fwd figures so all account transactions must be reloaded - there will be hundreds of millions - it just isn't feasible. Someone has clearly got the wrong end of the stick.

Also rounding on currency conversions is different so we wont be able to cross check balances in all cases!

All the screens look very different to teams are already making processing errors and it's going slowly.

>> It's one of your core systems and you have no decent lead and the supplier
>> is telling you what will or will not be done.
>>

We are a small division in one of the UKs largest banks. Very little core IT project support - usually here is some dosh got on with it. Job of sorting was given to senior Ops manager.


>> What a total nightmare. Even if everything goes perfectly it will still screw your operations
>> for a few days.
>>

Yes - there is no parallel running envisaged.

>> Who asked you to do this sense check?

My MD.

>>Tell them, immediately, in writing, everything you're not happy with. Because you've been asked to "sense check", thus any continuing nonsense is
>> happening on your watch.

Email sent last night.

Typed memo sent recorded delivery from home on headed work paper guaranteed before mid day to signed for at 11AM this morning.

>> But you certainly do need a data migration strategy. Whether or not it will be
>> automated.
>>

Trying to get hold of the supplier is difficult. They are all working from home and not returning my calls as I am not on their "who to deal with" list. Trying to get that sorted.


>> And you abso-bleedin-lutely need testing. Of everything.

Yes. Started that process already. Found out some of the interfaces to SAP have changed and the new system creates plain text batch files dumped in to a folder which they expect SAP to pick up! No encryption for financial transactions. I'm fuming!

The same goes for the interface to the core banking system - i.e. someone could be able to edit payment instructions before they are processed without an audit trail!!!

That has got to change!!! How other banks haven't noticed - I don't know!?

Speaking with MD this morning he is getting someone from head office IT projects involved - hopefully it's not too late!

This is going to be fun, NOT!


 Large Project Data Transfer - smokie
Seriously, it sounds to me like your job is finished and you've earned your corn, I'd just back out gracefully now if I were you and leave it to the "experts", else it could all backfire on you, or become an absolute millstone for you. Quit while you're winning!!
 Large Project Data Transfer - Zero
You have your IWMG* get out of jail free card signed, now its time to update your CV**

*It wasnt me guvner

**Not because you will be sacked because of this fiasco, but because there is a chance your employer may well be defunct shortly.

 Large Project Data Transfer - zippy
>> You have your IWMG* get out of jail free card signed, now its time to
>> update your CV**
>>
>> *It wasnt me guvner
>>
>> **Not because you will be sacked because of this fiasco, but because there is a
>> chance your employer may well be defunct shortly.
>>

Whilst we never know what’s round the corner, this is a small division of one of the UKs biggest banks. A lot of our pension savings hold these shares!

 Large Project Data Transfer - Zero
Isnt your divisions IT audited in any way by the parent company?

I'm sure they must be, its a requirement by many regulatory frameworks
 Large Project Data Transfer - zippy
>> Isnt your divisions IT audited in any way by the parent company?
>>
>> I'm sure they must be, its a requirement by many regulatory frameworks
>>

They are. The existing system passed with recommendations for some minor upgrades.

The new one should be procured in line with all company policies.

I have raised an alert re the weak process trail between our new back office system and the banks payments system. As far as I can see - this is from 1 day working on it - anyone can edit the file with a text editor and change the payment value, , payer details, sort code and account number or even drop in a New text file with details to be processed next batch run.

Christ knows what the better paid management have been doing on this project!?
 Large Project Data Transfer - zippy
Just had a delivery to my home.

Glenfiddich 21 Year Old Gran Reserva Whisky and a thank you note from the MD.

Very nice too!

At least there was no revolver included!

Back to the day job!
 Large Project Data Transfer - smokie
Send round a virtual dram to us all then!! :-)

Sounds like a good outcome.
 Large Project Data Transfer - zippy
>> Send round a virtual dram to us all then!! :-)
>>
>> Sounds like a good outcome.
>>

Cheers all, your advice and pointers and support is very much appreciated!

Latest Forum Posts