Tuesday, July 24, 2012

GC, Memory, and Multi-Threading in C#

GL to me, my 50th post and is the 1st technical post.
if ur a senior programmer u might be bored and r welcomed to comment.

lets start.
i have something to do around, say, 60 times. (usually 5000 time, *10 sec is 14 hrs)
linear, like for, takes say 60 * 10 sec =  10 mins.
but if i multi thread it, say 10 at a time in a good cpu i can get to 20 sec or less for all 10 threads, so where done to 2 min (or almost 4 hrs) - cool!
so in a basic way to do that would be that : http://bresleveloper.blogspot.co.il/2012/05/muti-threading-in-forforeach-example.html

now u should thing hey - says who that after all these threads are finished all this CPU usage and memory gets free? msdn says so : "For the majority of the objects that your application creates, you can rely on the .NET Framework's garbage collector to implicitly perform all the necessary memory management tasks. However, when you create objects that encapsulate unmanaged resources, you must explicitly release the unmanaged resources when you are finished using them in your application. "

so rule 1: if we dont use unmanaged code we dont need to worry about our memory. or better to say memory leak.

so when will i call the GC? according to msdn : "In almost all cases, you do not have to call this method, because the garbage collector runs continuously. This method is primarily used for unique situations and testing."

so rule 2: we do not call the GC
if ur really hot on using GC then google about "when to use GC C#"
if it solves you problem then either ur really 1 of the special cases (probably not) or you missed something down here.

so basicly calling thread and / or function that only use managed code, i.e. .Net classes and structs that do not implement IDISPOSABLE that we can be at ease that memory is managed well.

when do i need to start thinking?
in our basic case - i the thread gets an exception. for an infinite loop case u can put a timeout. but if the thread gets and exception it will probably get stuck and "eat" ur memory.
at the begining i tried to check each thread and NULL it and its container (List<>).

but that was wrong.
the good solution is a good OOP design where ur thread is wrapped in a T/C (try catch) so that if any exceptions occur it will end, and if it ends it gets collected
if you cant do that that Thread.Abort() would do it, and it will be collected the moment u leave the method or the moment you clear ur container, as long as there is no other place refering (pointing) it.

so to finalize that part - in managed multi threading make sure all ur thread gets finished safely, best with T/C wrap.

and when i use unmanaged code?
the very basic is that every managed class implement IDisposable, so call the Dispose() method (and /or Close() if needed, if its there u usualy need it).
Best practice is to use the using(){} wrapper wherever u can (yes, even if its "stupid" having 15+ using)
when having a class using unmanaged give it the IDSIPOSABLE and implement it calling the Dispose() of everything, same in methods, same in threads.
in classes its also good to use Destructors, usually u can just call Dispose() there.

conclusion: if u implement everywhere Dispose() and "freeing" all the references ur program should be fine.

if u still find urself with ur machine choking then here is a check list
1 - did i lose ALL the references everywhere?
2 - did i close ALL my streams/connection/anything with Close method?
3 - did i disposed every resource when not using it anymore, i.e. calling Dispose on everything that have dispose when i finished using it?
4 - are all my threads gets finished properly and safely?

*global instances are the dangerous part here, and it will be (harder work but) better to put every instance in its main/top method and pass it to all the submethods or create more classes for those cases.

if all these are checked than either u missed something (sry) or find ur Ex or comment me when u find the problem.
also a few interesting point:
- the part of the memory that gets "stuck" is the part that belong to Resources
- therefor the GC might keep all kind of ref-types to it
- usually the system will get slow becuz of the Resources becoming heavy and GC.Collect wont help
- basicly a well written class has a finalizer that is being called when the c# class no longer holds references so unless u use something u wrote, with classes that MS wrote u should be fine even without calling dispose (o.c. not recommended)

and to the one of most interest and i quote:
"Dispose() immediately releases the resources associated with your object. Yes, GC is non-deterministic, but calling Dispose() does not trigger a GC!"
so it ease the entire OS but does not release ur app personal memory.
from here:
http://stackoverflow.com/questions/7520881/is-it-bad-practice-to-depend-on-the-net-automated-garbage-collector

an excellent and simple article to understand the .NET GC and destrucors:
http://www.c-sharpcorner.com/uploadfile/chandrahundigam/understandingdestructors11192005021208am/understandingdestructors.aspx
another good summery:
http://stackoverflow.com/questions/4267729/what-happens-if-i-dont-call-dispose-on-the-pen-object/5555243#5555243

Monday, July 23, 2012

c# to MySql DateTime / TimeStamp Format

this is not about how to extract datetime formats from mysql, there r many examples AND if ur using C# / .NET then the VS knows the job and doesn't need ur help.

BUT! when i try to put MY datetime to the mysql  - nothing goes there!
well, that's cuz i'm not familiar with mysql datetime formats, and yes, i am not using SP, just simple queries.
1 - the format is yyyy-MM-dd hh:mm:ss, that is:
     string date = DateTime.Now.ToString("yyyy-MM-dd hh:mm:ss");
2 - TimeStamp: - on INSERT doesn't get value (null - reading-parsing) and wont accept 1970 and earlier
3 - and since any other mistake would NOT resolve in null but something unsolved by me, in an sql query u can only see them like that:
SELECT UpdateDate FROM franchise_information WHERE UpdateDate LIKE '% %';
for C# u'll need this:
MySqlDateTime d = r.GetMySqlDateTime(0);
anything else would give you EX : "Unable to convert MySQL date/time value to System.DateTime"
and that is cuz if u'll try d now u'll see "{0/0/0000 00:00:00}".

conclusions:
when inserting new stuff where there is a timestamp (on update):
1 - put a value
2 - don't call it
3 - call it safe

worst case scenario:
query = "SELECT * FROM table WHERE ID = " + ID;
try
{
    t = mysql.ExecuteSelect(query).Tables[0];
}
catch (Exception ex)
{
   
if (ex.Message.Contains("Unable to convert MySQL date/time value to
                            System.DateTime"))
    {
         string date = DateTime.Now.ToString("yyyy-MM-dd hh:mm:ss");
                 //could be either of these
         query = "UPDATE table SET UpdateDate = '1971-01-01'
                  WHERE ID = " + ID;
         mysql.ExecuteUpdate(query);
         query =
"SELECT * FROM table WHERE ID = " + ID;
         t = mysql.ExecuteSelect(query).Tables[0];
    }
    else
       throw ex;
}





Wednesday, July 18, 2012

Insert \r\n into MySql

replace \r\n to \\r\\n.
p.s. vs should know how to do it, make sure u dont have a running around '.

Tuesday, July 3, 2012

Parallel.For with/update List example, ThreadLocal Explained

FINAL CODE:
  Parallel.For(0, lCount, () => new List<int>(), (i, loop, lLocal) =>
    {
      lLocal.Add(i);
      return lLocal;
    },
    (lFinal) =>
    {
       lock (this)
       list = list.Union(lFinal.AsEnumerable()).ToList();
    }
  );


EXPLANATION:

did u read MS example and had a hard time? me 2 :)
http://msdn.microsoft.com/en-us/library/dd460703.aspx
so lets make it simpler:
1 - what we're doing is this
   for (int i = 0; i < nums.Length; i++)
      total += (long)i;

2 - the catch: http://msdn.microsoft.com/en-us/library/dd997392.aspx
3 - the solution - give the loop(s - i'll show u later) a local var to use during the iteration

4 - whats the new loop is doing :
Parallel.For<long>(   // <T> is the type that we are working with, here  
                            summing a long
      0, nums.Length, // begin-end iteration values,
                            i.e. "int i = 0; i < nums.Length;"
      () => 0,        //initialize your local state (var), out subtotal
      (j, loop, subtotal) =>
                      //is something like this public <T> DoWork(int i,  
                           ParallelLoopState, <T>)
                      //int i is the current i value in the loop
                      //ParallelLoopState - instance to break the loop.
                      //<T> - our working var, each DoWork gets it,
                           work on it, and return its like (or it)
      {
         
//since i am a lambda created, u dont need do new to my inputs,
                 nor declare my return
         subtotal += nums[j];
         return subtotal;
      },
     
(x) => Interlocked.Add(ref total, x) // type <T>
        // will be called after a set(!) of iterations completed.
);

great!
but now i wanna do this
   for (int i = 0; i < 10000; i++)
     list.Add(i);

which becomes this
Parallel.For(0, 100000, () => new List<int>(),// we want to work on filling a list
  (i, loop, lLocal) =>  // lLocal is our list, our local var
  {
     lLocal.Add(i);
     return lLocal;     // and therefor our return type
  },
  (lFinal) =>
Interlocked.Exchange<List<int>>(ref list, lFinal);
       // i.e. put lFinal in list, could also be just list = lFinal
);

 i got about half the numbers!?!
if u read the catch part u'll see they're talking about it.
lets put in the MS example this:
   (lFinal) =>
   {
      Interlocked.Exchange<List<int>>(ref l, lFinal);
      Console.WriteLine(lFinal.Count);
   }
our output will be something like:
XXXX // like 2856 
XXXX
XXXX
XXXX
usually between 4-6.
meaning Parallel does 4-6 threads, each updating list
so we need to create something thread safe AND unioning to our list
Interlocked is a special class that is making the exchange "atomic", as in atomic speed, so its what we want to lock "real time" something like our update to our list
so i got to this:

int usingList = 0;
Parallel.For(0, lCount, () => new List<int>(), (i, loop, lLocal) =>
{
   lLocal.Add(i);
   return lLocal;
},
(lFinal) =>
{
   while (true)
   {
       if (Interlocked.Exchange(ref usingList, 1) == 0)
       {
           list = list.Union(lFinal.AsEnumerable()).ToList();
           Interlocked.Exchange(ref usingList, 0);
           break;
       }
       else
           Thread.Sleep(3);
   }
   Console.WriteLine("done " + lFinal.Count);
});

luckly .NET have something that does this exactly
FINAL CODE:
  Parallel.For(0, lCount, () => new List<int>(), (i, loop, lLocal) =>
    {
       lLocal.Add(i);
       return lLocal;
    },
    (lFinal) =>
    {
       lock (this)
         list = list.Union(lFinal.AsEnumerable()).ToList();
    }
  );

P.S. - for this simple task i found out that this also works
Parallel.For(0, lCount, (int i) =>
  {
     lock (this) l.Add(i);
  });




How To: Not Using app.config

the answer: delete it from the proj tree (of from folder) and then from the .csproj file.
TA DAM!

the story?
well well, again MS don't REALY care about you.
why?
cause they can't put anywhere a little "don't use/create app.config"!!!

my app is using a dll that is importing a general .config file for all kinda stuff like log4net ect.

so i changed a FW in my proj and BOOM! no logger!
and no way from the VS(10) to remove it or when u press F5 u get "operation can not be complete".

same thing about converting a VS10 proj/sln to VS8 - 2 bloody lines in the .csproj file, can't thay handle it?!?