MEGAsquirt A place to collectively sort out this megasquirt gizmo

Small Software Project -> Logging!

Old 02-13-2008, 05:10 PM
  #1  
Elite Member
Thread Starter
iTrader: (3)
 
AbeFM's Avatar
 
Join Date: Aug 2006
Location: San Diego, CA
Posts: 3,047
Total Cats: 12
Default Small Software Project -> Logging!

Originally Posted by Joe Perez
cjernigan, I have been boosting, gently. I haven't yet installed EBC, and I've got my MBC bypassed, so I'm running on base can pressure- about 5 to 6 PSI. I haven't really romped on it much since the VE table still needs a good dyno session, but I have been boosting mildly. I'm really impressed by how totally seamless the transition into boost is with this setup relative to the EMU- no sudden jump in AFR, just a nice, smooth curve.
This brings in a nice side project I want to do, when I find someone who wants to do it for me. :-) I want to make a script for concatenating log files. MegaLogViewer has a nice autotune function (if you keep an eye on it), and I have log files at probably 6 different boost levels. I could do all my tuning in one go, averaged over a huge range of settings and with hours of log-time, but I want a way to string them together.

Any ideas? They are just .xls, and you have to do a small amount of tweaking (remove/renumber the manual mark/resets, and maybe the timestamps need a touch up.
AbeFM is offline  
Old 02-14-2008, 08:00 AM
  #2  
Junior Member
 
richard_ha's Avatar
 
Join Date: Jan 2007
Posts: 188
Total Cats: 0
Default

I've also been thinking about this. Essentially just creating lost of logs about 20 minutes long and being able to add them all together and create one megalog (think i've just come up with the program name) of hours worth of data to do a VE analysis with.
richard_ha is offline  
Old 02-14-2008, 08:03 AM
  #3  
Junior Member
 
richard_ha's Avatar
 
Join Date: Jan 2007
Posts: 188
Total Cats: 0
Default

All you have to do is open one file, and paste another on the end (not including the headers) it works fine.
richard_ha is offline  
Old 02-14-2008, 08:34 AM
  #4  
Elite Member
iTrader: (1)
 
richyvrlimited's Avatar
 
Join Date: Jun 2006
Location: Warrington/Birmingham
Posts: 2,642
Total Cats: 42
Default

Originally Posted by richard_ha
I've also been thinking about this. Essentially just creating lost of logs about 20 minutes long and being able to add them all together and create one megalog (think i've just come up with the program name) of hours worth of data to do a VE analysis with.
It's a nice idea, but I did a datalog of 30ish miles, (just from home to work) and then used the analysis on my desktopp PC (2.5Ghz dual core 2GB ram etc etc.) and it properly chugged along, took about 20 minutes to analyse the file.

There is a benefit to longer files, but only to a point.
richyvrlimited is offline  
Old 02-14-2008, 08:57 AM
  #5  
Junior Member
 
richard_ha's Avatar
 
Join Date: Jan 2007
Posts: 188
Total Cats: 0
Default

Really? My laptop did a 45 minute log in about 10 seconds
richard_ha is offline  
Old 02-14-2008, 09:53 AM
  #6  
Elite Member
iTrader: (1)
 
richyvrlimited's Avatar
 
Join Date: Jun 2006
Location: Warrington/Birmingham
Posts: 2,642
Total Cats: 42
Default

Originally Posted by richard_ha
Really? My laptop did a 45 minute log in about 10 seconds
10 seconds? did you log anything in that 45 min period
richyvrlimited is offline  
Old 02-14-2008, 10:35 AM
  #7  
Junior Member
 
richard_ha's Avatar
 
Join Date: Jan 2007
Posts: 188
Total Cats: 0
Default

yeh, 45 minutes worth of driving!

You might need to update your java installation or something?
richard_ha is offline  
Old 02-14-2008, 10:58 AM
  #8  
Elite Member
iTrader: (5)
 
Atlanta93LE's Avatar
 
Join Date: Sep 2006
Location: Marietta, GA
Posts: 2,195
Total Cats: 1
Default

Take a look at MegaTweak (I believe it is actually MegaTweak3000). It is essentially a standalone VE Analyzer that takes many logs as the input.
Atlanta93LE is offline  
Old 02-14-2008, 11:26 AM
  #9  
Boost Czar
iTrader: (62)
 
Braineack's Avatar
 
Join Date: May 2005
Location: Chantilly, VA
Posts: 79,484
Total Cats: 4,076
Default

Braineack is offline  
Old 02-14-2008, 11:32 AM
  #10  
Elite Member
iTrader: (1)
 
richyvrlimited's Avatar
 
Join Date: Jun 2006
Location: Warrington/Birmingham
Posts: 2,642
Total Cats: 42
Default

Originally Posted by Braineack
wtf?

I'm well aware of it's inbuilt filtering, but the filtereing doesn't really have anything to do with the thread....
richyvrlimited is offline  
Old 02-14-2008, 11:35 AM
  #11  
Boost Pope
iTrader: (8)
 
Joe Perez's Avatar
 
Join Date: Sep 2005
Location: Chicago. (The less-murder part.)
Posts: 33,015
Total Cats: 6,587
Default

Surely even the MS-II dweebs can use Autotune?

Joe Perez is offline  
Old 02-14-2008, 12:37 PM
  #12  
Elite Member
Thread Starter
iTrader: (3)
 
AbeFM's Avatar
 
Join Date: Aug 2006
Location: San Diego, CA
Posts: 3,047
Total Cats: 12
Default

Joe, we can't just be expected to go clicking things at random..... I was a big autotune fan, then I just somehow never used it. :-) The point is I have a lot of logs already in exsistance I want to use - covering a lot of ranges. Also, have you seen the weird things auto-tune/auto-analyze will do sometimes?

Nonetheless, there's a few reasons to do this: Also, it makes a huge log file so I can look for certain 'events of interest'. So in an ideal world, it would bring the marks and resets with it.

Originally Posted by richard_ha
All you have to do is open one file, and paste another on the end (not including the headers) it works fine.
Does it? It didn't for me! There really isn't much header, but skipping that line, it seems to get mad about the RESETs and MARKs not being sequential.

Originally Posted by richyvrlimited
It's a nice idea, but I did a datalog of 30ish miles, (just from home to work) and then used the analysis on my desktopp PC (2.5Ghz dual core 2GB ram etc etc.) and it properly chugged along, took about 20 minutes to analyse the file.

There is a benefit to longer files, but only to a point.
Oh, I'm sure there's a point to longer files, more averaging. I hate to see only 15% of my cells change, also.

And, yeah, my logs, even hour long drives, take a few seconds to run (on a decent quad core machine), but even on work's crappy laptop it still is a matter of a couple tens of seconds for a typical 20 minute log at most. You need to figure out why yours is running slow, something is wrong.

Originally Posted by Atlanta93LE
Take a look at MegaTweak (I believe it is actually MegaTweak3000). It is essentially a standalone VE Analyzer that takes many logs as the input.
Oh! There's an answer I like. I couldn't find it on ten-seconds-at-google, but I'll keep digging.
Interesting, DIY says it's "outdated, use MLV now"
AbeFM is offline  
Old 02-14-2008, 01:01 PM
  #13  
Boost Pope
iTrader: (8)
 
Joe Perez's Avatar
 
Join Date: Sep 2005
Location: Chicago. (The less-murder part.)
Posts: 33,015
Total Cats: 6,587
Default

I just stitched two logs together in Excel and MegaLog digested the result with no complaints.

First, I made sure that there was only one header, by copying and pasting only row 3 and down from each subsequent log onto the end of the first log.

Next, I searched for "mark" and deleted all rows containing one.

Last, I found the point at which each new log segment began (I'd previously highlighted the last line in each segment before pasting the next segment in) and adjusted the time (column 1) for each log segment by adding the stated time to a constant which was the last stated time of the previous log. (I didn't touch the 0-255 counter, only the realtime.)

Then I saved the result as text (tab delimited), changed the extension to .xls, and loaded it into MLV.
Joe Perez is offline  
Old 02-14-2008, 05:10 PM
  #14  
Junior Member
 
richard_ha's Avatar
 
Join Date: Jan 2007
Posts: 188
Total Cats: 0
Default

What version of MLV are you using? As i said it worked fine for me just adding on on the end (minus the first two lin)
richard_ha is offline  
Old 02-14-2008, 08:02 PM
  #15  
Elite Member
Thread Starter
iTrader: (3)
 
AbeFM's Avatar
 
Join Date: Aug 2006
Location: San Diego, CA
Posts: 3,047
Total Cats: 12
Default

I asked the guy who writes MLV and he said he gets the request a lot. Mine's the newest, generally, since it autoupdates. I'd like to make a script/macro to do it, though my excel skills aren't all that. Ideally, re-enumerating the marks, but that's likely too much to ask for. It's just kinda a chore to do it by hand!

-Abe.
AbeFM is offline  
Old 02-14-2008, 08:24 PM
  #16  
Junior Member
 
richard_ha's Avatar
 
Join Date: Jan 2007
Posts: 188
Total Cats: 0
Default

Try opening this in MLV.
Attached Files
File Type: zip
MegaLog.zip (379.8 KB, 37 views)
richard_ha is offline  
Old 02-14-2008, 10:18 PM
  #17  
Elite Member
iTrader: (11)
 
elesjuan's Avatar
 
Join Date: Jun 2007
Location: Overland Park, Kansas
Posts: 5,360
Total Cats: 43
Default

Originally Posted by richyvrlimited
It's a nice idea, but I did a datalog of 30ish miles, (just from home to work) and then used the analysis on my desktopp PC (2.5Ghz dual core 2GB ram etc etc.) and it properly chugged along, took about 20 minutes to analyse the file.

There is a benefit to longer files, but only to a point.
I did about 7 hours of data logging when I first got mine going, all in one shot. My logging/tuning computer is a Sony Notebook, Pentium 3 1.0ghz, 328mb ram (piece of **** sonys...) running XP Pro. That took me all of maybe 14 - 17 minutes to process. Dunno why yours takes so long.
elesjuan is offline  
Old 02-14-2008, 10:36 PM
  #18  
Boost Pope
iTrader: (8)
 
Joe Perez's Avatar
 
Join Date: Sep 2005
Location: Chicago. (The less-murder part.)
Posts: 33,015
Total Cats: 6,587
Default

Originally Posted by elesjuan
(piece of **** sonys...)
Say what you will, but the Vaio TX and TZ-class machines are just about the awesomest notebook since the Tandy 100. Sure they ain't the fastest thing in the universe, but they're smaller than hustler's dick and yet they still have built-in optical drives.
Joe Perez is offline  
Old 02-14-2008, 10:57 PM
  #19  
Junior Member
iTrader: (1)
 
grippgoat's Avatar
 
Join Date: Apr 2006
Location: Kirkland, WA
Posts: 239
Total Cats: 0
Default

I should be able to whip this up in C# super fast.

Starting.... NOW!

EDIT: what do marks look like in the log?

-Mike
grippgoat is offline  
Old 02-14-2008, 11:31 PM
  #20  
Junior Member
iTrader: (1)
 
grippgoat's Avatar
 
Join Date: Apr 2006
Location: Kirkland, WA
Posts: 239
Total Cats: 0
Default

Takes a list of log files on the command line. Strips out marks. Adds an offset to the timestamp on later logs. Dumps the output to the console, so you have to pipe it into a new file.

Requires .NET 2.0 Runtime

Zip:
MegaLogCombine.zip

Code:
Code:
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;

namespace MegaLogCombine
{
	class Program
	{
		static void Main( string[] args )
		{
			double timeStampOffset = 0;
			double currentTimeStamp = 0;

			int iLog = 0;
			int fieldCount = 0;

			foreach ( string arg in args )
			{
				string[] lines = File.ReadAllLines( arg, Encoding.ASCII );
				for ( int iLine = 0; iLine < lines.Length; ++iLine )
				{
					string line = lines[ iLine ];

					// strip out the first two lines of all logs besides the first
					if ( iLine < 2 )
					{
						if ( iLog == 0 )
							Console.WriteLine( line );

						continue;
					}

					// Don't want mark lines
					if ( line.Contains( "\"MARK" ) )
						continue;

					string[] fields = line.Split('\t');

					// don't do empty lines
					if ( fields == null || fields.Length < 1 )
						continue;

					if ( fieldCount == 0 )
						fieldCount = fields.Length;
					else if ( fieldCount != fields.Length )
						throw new System.FormatException( String.Format( "Found {0} fields, expected {1} fields, on line: {2}", fieldCount, fields.Length, line ) );

					double timeStamp;
					if ( !double.TryParse( fields[0], out timeStamp ) )
						throw new System.FormatException( String.Format( "Failed to parse timestamp from line {0}", line ) );

					currentTimeStamp = timeStamp + timeStampOffset;
					fields[ 0 ] = currentTimeStamp.ToString();

					StringBuilder builder = new StringBuilder();
					for ( int i = 0; i < fields.Length; ++i )
					{
						if ( i > 0 )
							builder.Append( '\t' );

						builder.Append( fields[i] );
					}

					Console.WriteLine( builder.ToString() );
				}
				timeStampOffset = currentTimeStamp;
				++iLog;
			}
		}
	}
}
-Mike
grippgoat is offline  

Thread Tools
Search this Thread
Quick Reply: Small Software Project -> Logging!



All times are GMT -4. The time now is 04:17 PM.