Writing Contents fo Data to CSV updating each each row individually

Hi,

 

New to forum so if anything contravenes forum etiquette let me know.

 

Using automate version 11.0.2.22.

I have created many jobs which write the dataset from an SQL query to a csv, and first time I've had this issue.

It looks like it is writing each record out individually (see screen shot) as it updates the row variable which is obviously taking for veer with 150k records.

Anyone know how to avoid this?

 

Thanks,

 

Dave

 

Hi Dave,

Though you are seeing each row being written within Task Builder, you have to keep in mind that Task Builder is also slower execution than execution from the task/agent service. Have you tried executing the task outside of Task Builder to gauge the execution time? If not, I recommend trying that first. I create a lot of tasks that, for debugging purposes, run a lot slower in Task Builder which is intentional but does not reflect actual execution speed outside of development.

Regards,

Leonard

Hi Leonard,

 

I have run it in batch and it is amlost 2 hours now and it's still not finished so I don't think that's the issue

 

please see screen shot

Hi,

 

It has been running in batch for over 2 hours, on step 15 which is writing to the csv from dataset

Hi Dave,

Our actions that write from datasets do proceed row by row.  This does mean that the more rows and columns in a dataset the longer this process will take.  Do you know approximately how many rows and columns are in the dataset for the task in question?

I know this is an old post, but will HelpSystems address the slowness of dataset to csv? I just ran a job in Automate writing 192,000 records from SQL Server table to a csv file. It took 26 minutes. I then tried the same on Winautomation and it took literally 5 seconds. I then tried on UIPath Community and it took 5 seconds there as well. Please HelpSystems, fix this issue as ETL is a big part of RPA. Thanks

I agree that AutoMate is very slow at writing large datasets to csv files... and does depend on row/columns etc,  
The work around I've used for over 10 years is to use basic script.  This requires some coding.  

http://forums.networkautomation.com/forum/messageview.cfm?catid=41&threadid=9191

My work around is to use an exe generated by Winautomation. I then use that executable inside a larger Automate workflow. Works fine. I just wish I didn't have to use 2 products.

*WEB*Hello,

 

We have just released our newest version of Automate Desktop v11.3 has improved the performance of the File System (Dataset to CSV) activity, Issue #25029. You will need to login into your account page to download the .exe as we no longer provide direct links for the download.

 

I have added a link to the portal below;

https://community.helpsystems.com/support/

 

Below is a link to the change log;

https://static.helpsystems.com/releasenotes/Content/_ProductPages/Automate/AutomateAMEnt.htm

 

Upgrade Process:

1. Uninstall Automate Desktop from Add/Remove Programs (Do not add a check for any of the check boxes when prompted --> Select Next)

2. Make sure the Automate Desktop folder under program files C:\Program Files has been removed (If you are running the 32bit version you will need to check under the X86 Programs files folder)

3. Install our latest Version

Thanks Justin....huge improvement...dumped 250k records in seconds

Hi!

What about writing datasets to Excel?

It's a similar action and is used widely, but it wasn't improved in 11.3. When do you think to reach this too?

Hello Alex,

We are aware of the performance time when using large datasets being written to excel. We hope to improve the time in future versions of the software.

Currently, for a faster execution, it is recommended to use a Powershell or Basic script, for example, which writes in a much faster time frame, and to run the script from AutoMate.

Hi&nbsp;<br> <br> Do you have a example of&nbsp;Basic script to export from a dataset to csv or Xls file?<br> <br> Thank you very much