A Reader’s Suggestion for DEEP VALUE COURSE

I have recently received many suggestions to improve this course. There are almost 600 people in our group with many different backgrounds and experiences.  There may be advanced students who wish to discuss current case studies (Why is DLX, Deluxe, not a value trap like Radio Shack?) or potential investments or subjects into greater depth.  This course planned to follow Deep Value (book) while digging deeper into the footnotes without preconceived notions.  If, for example, we read about Buffett’s transition from net/nets to franchises, we will look at franchises but not focus on them. The point is to give students background to understand the distinctions. However, this may be too basic for many of you.

My goal is to make this a learning community. One idea would be to set up another blog (volunteers?) to discuss various ideas if there is enough interest. I won’t give out anybody’s email without their permission, but if a group of students wanted to dig deeper into various subjects, I am happy to provide a link to this new group or discussion area.  I will wait until I hear your feedback.

For example, a reader/student went into vast effort to provide feedback and suggestions.  Dr. K (my nickname for this reader) might be an excellent leader to develop a new discussion blog?  Below are Dr. K’s emails and links.  I will post your suggestions.


John; This is Dr. K following up from our phone conversation. So far I am very frustrated with the deep value course and in the following series of emails I will explain why.

Why I hate mechanical investing and problems with back-testing

  • First of all Seth Klarman in his book (Intro xvi, p.13, p.16-18, p.151, p.162) discusses the folly of searching for the holy grail “mechanical” formula for investing success. BG in The Intelligent Investor p.38-46 and p.194-195 also says mechanical formula investing is self-defeating. It contradictory that we are reading about mechanical formula (Toby Carlisle’s books). Mechanical formula (Toby Carlisle and Joel Greenblatt’s 2nd book) come off as lazy, naïve, and immature to me.
  •  Seth also is not a fan of Wide Moats (see p.32-34, p.93) but I guess that’s OK but it’s very confusing how we are jumping around from one investment ideology to another!! Seth also does not think much of EBITA (see p.71-78) while you seem to mention it on the front of your blog home page in a link at the top.
  •  Now go to http://falkenblog.blogspot.com and read as many past articles as you can. You should also get his first book: “Finding Alpha.” There is no need to read the rest of his books. In that book especially key in on p.115-116 “Geometric vs. Arithmetic Averaging”, p.116-117 “Survivorship Bias”, Ch. 6 p. 113-125 “Is the Equity Risk Premium Zero?” especially read p.121-123 “Transaction Costs” , p.47 “the Size Effect” and p.48 “Delisting bias.” These are just some of the many reasons why mechanical back testing is a dead-end path to investing success.
  •  And/or you can go to http://www.efalken.com and watch the Finding Alpha videos. This should take 8 hours but it’s good if you are short on time or you can do both. I also have many of the papers he has written or mentioned. I can send them to you upon request.
  • Next go to www.davidhbailey.com.Click on the Mathematical Investor blog and read all of the articles. Now see that attached papers he wrote along with Campbell Harvey.
  • Go to www.numeraire.com and read all of the links at the top. In the search site map section click on the article about why screening is not valuation.
  • Now go to www.numeraire.com/download.htm . Read every article in the Research Related Letter section. These are short but notice whom he is critical of. Now read all of the articles in the Research Notes section. Especially read “What is Circular Reasoning” which will explain why most stock screens and mechanical formulas (Toby Carlisle, Joel Greenblatt) are garbage! Make sure you click on or go to all of the links in this article. Notice the lists of various forms of logical errors. Notice the cross correlation with behavioral finance! These logical error list need to be studied in greater detail!! Also especially read “What is Economic Simultaneity?”, “A History of the Size Effect” and notice that on p.4-5 he lists some of the variables that are and are not circular!!, “Evolution of Stock Picking”, “Visual Detection of Circular Reasoning” (it’s vital you understand this) and finally “Fatal Summary.” (it’s vital you understand this also) Read all of the articles in Research Presentations and in Research papers (especially “Circling the Square” and notice on p.8 he gives the return formula and it’s vital you understand this).

You should a somewhat better idea of why mechanical investing is not scientific. You should not trust academic research as well as a lot of p-side research! I am not impressed with Toby Carlisle and Joel Greenblatt’s second book. I have a good feeling that many people in this course and google group are more advanced then you are aware of and feel the same way. I can tell you that I have been reading many academic papers over the years and would attach them when I tried to apply for a research job and never got any positive feedback from attaching them!! In addition Alpha Titans such as Seth Klarman, Peter Lynch, Warren Buffett, Ben Graham were not fan of this type of investing approach.

Let me know when you have read this stuff. I think you need to read and understand them otherwise you are not fully grasping how difficult value investing really is!! Keep in mind I don’t fully understand everything in this material but I know enough to not be impressed by mechanical investing research garbage studies!!

Most Factor Models that Explain Returns are False _ 1

Anamolies Don’t Do as Well after Publication

A Skeptical Appraisel of Asset-Pricing Tests _ 3

Backtest Overfitting

Significance Testing Issues for Empirical Finance 5

The Probability of Backtest Overfitting 6

Factor models to sensative to the time period tested 7

Email #2 about Net Nets.

You mentioned www.oldschoolvalue.comon your site in the resource section. I like this site for the free screens and the blog articles which do a good job of teaching quality investment theory. I am not too fond of his software program and he raised the price and his spreadsheets do not include critical off-balance sheet accounting adjustments (I will explain the New Constructs platform in a later email). Jae Jun thinks his spreadsheet program is better than it really is. (in my opinion).

Go to the -VeEV, NNWC and NCAV screens. Some of the stocks duplicate themselves. Now go to Ben Graham Net Net Stocks and a 7 Step Checklist to Make Money with Net Nets . Notice on p.7 Jae says 99% of Net Nets are useless and on p.9 he does not like Chinese ADRs. I think he also does no include financial companies, REITs, CEFs, Shells etc.

Now go to www.grahaminvestor.com.In the screens link at the top go to the NCAV Stocks (Shares Outstanding) screen. Notice it does include Shell companies and Chinese ADRs but only companies listed in the USA. (no Japanese, Canada, Australia, etc.) The next screen is NCAV Stocks (Float). I am not sure what the advantage of this screen is. It only seems to leverage the Current ratio. I have never seen anyone else mention using Float instead of Shares Outstanding instead. This site also gives: NCAV Stocks (Shares Outstanding, new) and NCAV Stocks (Shares Outstanding, new, no Chinese). I sent an email to the people that run this site about the screens but they never got back to me. I would not blindly trust that these companies are true Net Nets. You should verify the numbers yourself.

Go to http://www.netnethunter.com/my-ncav-investment-scorecard/. Does this guy give you the impression that he is a fan of running a mechanical screen and blindly following it?? Notice that he does analysis on the company’s “Burn rate.” Where did Toby Carlisle mention this?? Make sure you read all of the articles.

Now for the subscription. He does screens for Japan, Australia, Canada. The Japanese financials he gets are written in Japanese and gets the aggregated data from Business week.

A good assignment would be to figure out where for free or how you can screen for Japan, Australia, Canada or anywhere else other than the USA. Given the above two mentioned sites we could also for a day analyze (or for a week) every company of the list in greater detail and not do stupid mechanical investing like it has been suggested so far!!

I think I read that www.gurufocus.com has some Net-Net screens for a subscription rate but I don’t know if they are any good or not.

Another reason why Net Net studies and papers are flawed is that they don’t account for Survivorship bias and delisting bias in the historical database the study was conducted from. It’s quite possible that a company could be a “quality Net-Net” from financial standpoint but if the trading volume is too low then the NYSE could delist the company and then the stock loses 90% of its value as it goes from being a listed company to an unlisted (or OTC) company. Most of the deep value research does not discuss how to account for this and how to follow and trade these OTC companies!!

Here is email #3 about Wide Moat Investing.

You mentioned Pat Dorsey and his books. (see The Five Rules for Successful Stock Investing and The Little Book that Builds Wealth).

  1.  I met Pat Dorsey at a CFA Rochester, NY meeting while he was working for Morningstar. Morningstar has The Stock Investor newsletter which gives coverage of about 150 companies Morningstar believes are “Wide Moat” companies.
  2.  You mentioned Bruce Greenwald’s book and presentations about this subject.
  3. Now go to www.oldschoolvalue.com/blog/tutorial/this-is-how-buffett-interprets-financial-statements. This is a good summary article of the book “Warren Buffett and the Interpretation of Financial Statements” by Mary Buffett and David Clark. The authors also wrote “Buffettology” and a workbook about this book.
  4.  Also see What Gross Margins Can Tell You About a Company’s Economic Moat by Old School Value.

The assignment here is using these various books, articles and presentations someone subscribe to Morningstar’s Stock Investor newsletter, tell us the list of the 150 companies Morningstar list for “Wide Moats.” Then once we have this list we all go through each company and analyze and verify why these companies are indeed “Wide Moat” companies. In a presentation that you posted recently by WB he mentioned that there are no “Wide Moat” companies in Japan. I suppose we could locate “Wide Moat” companies in countries outside of the USA.


This is email #4 about Special Situation investing. You mentioned Seth Klarman’s book. In the second half of that book he gives various “Special Situation” opportunities. Joel Greenblatt’s first book also was pretty good (but  now a little out of date while his second book was a bunch of mechanical garbage).

a. The absolute best and easiest site to locate “Special Situations” is Spinoff Monitor – Actionable Opportunities in Special Situations: Spinoffs, Bankruptcy, Restructurings . Notice on the right hand side a very easy list of all of the various situations!! I am very interested in exploring this area of Investing!! Their is no analysis on this site.
b. Other sites that list Special Situations include:
c. You mentioned www.distressed-debt-investing.com in your resource section on your blog. Someone also attached Stephen Moyer’s book about Distressed Debt investing. My advice here would be to stay far away from this area of investing until the investor has more experience under your belt in analyzing distresses equities first. That website and book is very complicated as it requires specialized knowledge of Bankruptcy law, Quant Credit Modeling and simulation. You should not really list this site on your blog unless you make it clear to people this area of investing is not for beginners! It’s for advanced investors!!!!!!!!!!!!!!!! These resources are not written in a way that is easy to learn or read!!
d. There are many sites on the web for following the Insiders. I think I read that www.gurufocus.com gives you coverage of Insiders. Many sites are for free. The absolute best book on Insider Buying is “Investment Intelligence from Insider Trading” by H.Nejat Seyhun. It should be noted that Seyhun’s database goes all the way back to 1975 while I have yet to see any website that goes back that far for a affordable price.
Now here is what I think is going on with Spinoffs. Suppose you have XYZ company with a consolidated financial statement consisting of three divisions: Division A, Division B, Division C. So Consolidated XYZ = [A + B + C]. Now suppose Division B will be Spunoff. Now we have two companies: Parent Company = [A + C] and the Spunoff Company = [B]. I think what is going on here is that when this proposal takes place it won’t occur for a 90 day period so therefore their is a 90 day period where these two companies’ financial statements won’t be in the various databases. Instead Consolidated XYZ = [A + B + C] will still be in the database. I THINK WHAT YOU NEED TO DO IS RECONSTRUCT THE FINANCIAL STATEMENTS SUCH THAT YOU CAN ANALYZE PARENT COMPANY = [A + C] AND SPUN COMPANY = [B] AS TWO SEPERATE COMPANIES. DO NOT ASSUME AS MECHICANICAL INVESTORS DO THAT THE SPUNOFF COMPANY IS THE GOOD DEAL AND THE PARENT COMPANY IS THE BAD DEAL. EVEN IN JOEL GREENBLATT’S FIRST BOOK SOMETIMES THE PARENT COMPANY IS WHAT HE PURCHASED AND SOMETHIMES THE SPUN OMPANY WAS THE BETTER DEAL. I think Joel did an OK of explaining what was going on in his first book but he was not always clear about the timeline of events for how to follow a typical Spinoff situation.
As for the various other types of Special Situations (except for Merge Arb) their is almost no analysis or coverage on how to follow these events!!

Email #5 about “Expectations Investing.”

  1. Go to www.expectationsinvesting.com. Make sure you read “Expectations Investing (2000)” by Alfred Rappaport and Mchael J. Mauboussin and “Creating Shareholder Value (1998)” by Alfred Rappaport. THESE ARE THE ABSOLUTE TWO BEST BOOKS I HAVE READ THAT CLEARLY EXPLAINED WHY CONVENTIONAL ACCOUNTING IS FLAWED AND THE DIFFERENCE BETWEEN ACCOUNTING VALUE AND ECONOMIC VALUE CREATION!! I can’t go into two much detail here but make sure you read and understand every Tutorial on the site!! In a recent post your blog about Enron I think you were trying to highlight the concept of Incremental Capital expenditure. This and Incremental Working Capital expenditure are clearly taught in these two books. These books also do a wonderful job of explaining the underlying drivers of Economic Value creation! You want to understand these spreadsheets in detail!

The assignment here is that New Constructs (See links below) does about 22 various off-balance sheet adjustments. Learn these adjustments and modify the Expectation Investing sample spreadsheets. Know how to do this for companies that New Constructs does not cover. (such as REITs, MLPs, Net Nets, Special Situations, Japan and non-USA stocks). In the case of non-USA companies the accounting conventions would need to be researched.

I think these spreadsheets do such a good teaching job of explaining things!!

(Not posted here-I couldn’t open the zip file)

  1. Michael’s 2nd book “More then you know” and his 3rd book “Think Twice” are gems that do a good job of summarizing what is going on in behavioral finance and must be read. I have many of the papers he mentioned from these books.

30-00 Part 2

The-Best-Primer-to-Valuation-Multiples Part 2

Trouble with Earnings & PEs

Cash Flow vs. NI

Counting what Counts

Financial Ratios

Financial Ratioswheres the bar ROIC

CommonErrors in DCF Models

Decoding Wall Street Propaganda ( A MUST READ!)


Do Investors see through Reported Earnings




32 responses to “A Reader’s Suggestion for DEEP VALUE COURSE

  1. Canadian Content

    I think Dr. K needs to examine how critically and accurately he reads peer-reviewed, published research. Survivorship and delisting biases are easily, and consistently, corrected for. In his book, Quantitative Value, Wesley Gray details his process for eliminating these bias. These authors, (Montier, Carlisle, Gray, etc.) are not dummies, and they are not amateurs. I’d recommend Dr. K give a closer read to the methodology that is used to create the datasets of these popular studies, lest he continue to throw the baby out with the bathwater. Blog articles are not the same as peer reviewed, published research. The standards are much, much different.

  2. Canadian Content; I stand corrected if Quant Value does correct for these biases. However delisting and survivorship bias are still two problems that plaque most quantitative research in the financial field.

    No I don’t think James Montier is a dummy and probably is not guilty of these bias but I do feel that Eric Falkenstein does an outstanding job in his 1st book of pointing out these biases.

    I also feel that Numeriare and David Bailey do an outstanding job of describing the various problems of back testing in the financial field.

    All of the above mentioned papers should cause you to question the validity of most quant research in the financial field. (as far as back testing goes)

  3. Wow. Sounds like a deep value stock kicked Dr. K’s dog or something. I’m not sure what to make of this so i’ll leave my comment at that, and to cast my vote to have our lessons continue to follow the Deep Value book. Intuitively to me the process/rational makes sense.

  4. Canadian Content

    Dr. K, thanks for sharing your thoughts and those valuable resources. I look forward to having some time this long weekend to jump into those (many!) papers you posted. And I’m especially grateful for the spin-off resources you posted, as that is also an area of interest of mine, and one in which I’m working to gain greater knowledge.

  5. Dr K,

    I can understand your frustration.

    I think Toby’s book “Deep Value” does not add very much. It’s well written and it might serve to get people enthusiastic about investing. However, everything he wrote in that book is covered elsewhere.

    On another point, I don’t think it’s fair to suggest that Mr Graham was against mechanical formulas. Your citation to Intelligent Investor is quite correct, but Mr Graham was speaking there primarily about the theories for predicting overall market levels. I wouldn’t take that as suggesting that he wasn’t in favour of mechanical formulas for selecting individual stocks. Indeed, most of the chapter in Intelligent Investor on stock selection for “enterprising investors” is devoted to just such approaches. Further, in one of the last articles Mr Graham wrote (I don’t have it with me just now – it was republished in Janet’s Lowe’s book on the Rediscovered Ben Graham) he suggests a mechanical formula for laymen (medical practitioners, most of whom are inept on financial matters) based on two simple factors – low P/E ratio and low debt.

    On the other hand, I quite agree that mechanical formulas can’t be applied slavishly for some of the reasons noted in your email to John. In particular, financial data available from most published sources, at least in my experience, needs to be “cleaned up”. As an example, earlier this morning I started to look at a stock which hit a new low yesterday – our local financial press reported the P/E at 18.9x; a major reporting service had the P/E at 5.6x. Which is correct? Well, you have to look at the financial statements to work it out for yourself!

    As it happens, my own portfolio is partially based on a mechanical formula. Essentially, I trawl through the stock exchange listings every day looking for opportunities (eg, based on large price declines, bad press reports, etc, etc). I try to find 5 companies to invest in each quarter. To the extent I can’t find 5 per quarter, I make up the difference by buying “magic formula” type stocks (based on my own calculations of EV/EBIT & ROIC, taking into account post balance date events, etc).

    On the whole, I think John is doing a great job in terms of showing people how they can learn about value investing. It’s a worthwhile effort, although I don’t think the “Deep Value” book advances the chain much.


  6. 1. @canadiancontent is 100% correct – both Joel Greenblatt and Tobias Carlisle have gone to extreme and repetitive lengths to incorporate things like survivor bias into their research. Greenblatt reportedly spent $30m in building the infrastructure to even run the basic magic formula funds. Its laughable to think he didn’t think about the biases mentioned.

    2. Using inversion, why do simple screens like EV/EBIT work in REVERSE as well? If earnings yield screens added no value – that is, it provides no way to distinguish between companies that ‘outperform’ or ‘under perform’, then why does screening for high EV/EBIT result in extreme under-performance? Why does ranking stocks by EV/EBIT into quintiles or quartiles result in a dramatic difference in performance – even if flawed datasets are used? This shouldn’t happen if the metric was 100% circular or completely random.

    3. If Ben Graham – inventor of the net-net, the teacher of pupils like Walter Schloss and Irving Khan – wasn’t a quant, then maybe you are talking about a different Ben Graham

    I think Dr K has missed the forest for the trees. No one is saying basic screens are a substitute for valuation or hard-work. No one is saying that screens find every single undervalued stock in the world. Not one research report says that for a stock to be cheap it must be on a screen.

    What people are saying is that such screens are a fertile, but not only, hunting ground for finding POTENTIAL bargains. Using such a basic method, will outperform ON AVERAGE and over a long time period a passive approach (i.e. an index) that doesn’t even take into account what value-investors think to be true about basic valuation or psychology.

  7. Dr. K, did you apply for a job with Joel Greenblatt or Toby Carlisle and not get hired? Although your post seems to contain some very valuable information, you present it in a very pissed off manner. And no, not all of us are advanced when it comes to value investing, so stop speaking for everyone. You ‘appear’ to have quite a bit of knowledge, so why you’re not running your own blog or class to help others instead of trashing someone’s else’s attempt I don’t know. I do find interesting though how you trash mechanical investing, even citing Ben Graham as proof. Yet further into Deep Value Toby Carlisle reveals, and he’s not the only one to bring this up, the interview with BG later in his life where BG says were he to do it over again he’d stick to a mechanical method. It makes me wonder if you even read Deep Value before you decided to trash it. Which leads me to another point…since you’re so busy trashing the work of people with proven investment track records, where’s your book citing your methods for investment along with your validated results over at least a decade or two? Again, thanks for the links, but my advice…change your attitude or go somewhere else.

    Chad Green

  8. I appreciate your skepticism Dr. K.

    Simply stated, risk is not equivalent to volatility. So we already agree that at least half of the academic work in finance is not useful.

    There is a dangerous assumption in backtesting. The implication is that the strategies which yielded the largest returns in the past will yield the largest returns in the future.

    I think this is unscientific for at least 2 reasons:

    1. Sample size
    ->If I only care about beating the market over the next 10 years, then the prior 40 years are unlikely to be a large enough sample size. (especially if maintaining a concentrated portfolio)

    2. Adaptation
    -> if net nets and P/E multiples work for for 50 years, they might not work for the next 50 years because people might start using those techniques.
    -> business models and accounting methods change over time, therefore the effectiveness of mechanical techniques will vary as well.

    If we start by avoiding this naive assumption, then we can view the works of Gray and Carlisle in a different light. It raises some questions:

    1. If an investment strategy performed poorly in the backtest, why do you think it will work in the future?

    2. If an investment strategy performed well in the backtest, what does that tell you about this strategy in the future? What might it reveal about new strategies which have not been tested in real time?

    3. Some metrics like EV/EBIT perform better in the backtest than other measures. Does this mean that it’s actually superior? Or are the other techniques useful for screening?

    4. If an investment strategy works in a backtest, can we expect an emotional human to execute this strategy?

    Mechanical tools can be used directly as an automated strategy, or they can be used as a screening tool, or not at all.

    I think there is merit in all of these uses, and it’s up to each investor to decide.

    Dr. K, do you have a more direct criticism of the backtesting done by Gray / Carlisle? It’s hard to understand exactly where you’re coming.

  9. Dr. K, do you mind sharing the zip file here for our learning. Thanks!!

  10. My vote: Keep going with the DEEP VALUE Course. Great job John! Appreciate all the effort you put into it A LOT so far. Take care.

  11. Thank you John for your effort, I highly appreciate it!!

    @Dr. K:
    You seem to know a lot. Start your own blog! 😉

    John’s effort is a huge gift to the community,
    expressing frustration in that way is rude.
    And I do not share your frustration.

    There a few value fund managers in my country which are pretty successful
    with a quantitative approach.
    Of course compustat accounts for the survivorship bias!!!
    I checked it myself (Enron).
    Btw. In my opinion most of the academic research is useless.
    Often it is research on single factors.
    Who cares if the some ratio has a bad explanatory value,
    when it is a huge value-add if added to a multifactor model??

  12. Jae from Old School Value here.
    Thought there was something up with a few links coming my way.

    But I had to laugh. Dr K is certainly entitled to his opinions about Deep Value, Greenblatt, Old School Value, gurufocus, grahaminvestor but at the end of the day, it is just one person’s opinion out of thousands of happy readers/users/investors.

    And yes, I do think my valuation spreadsheets are awesome. Always working to improve it and currently working on an online version 🙂

  13. First of all, my thanks and compliments to John for assembling the materials and arranging this course.

    Secondly, I would be more than happy to moderate an “advanced” or hacker version or discussion group of the course. Perhaps the (non enforced) price of entrance would be one cogent idea or discussion of one of the techniques of getting alpha. (For the uninitiated, alpha is the extra return over a benchmark due to some skill set.)

    Now on to Dr. K. Well where to start? Dr. K seem to have done a fair amount of reading but might I suggest a caution and some humility, both as an investor and an analyst. I and all humans have this propensity for self-delusion that is universal and incredibly strong, and frankly it get me through the day.

    Presumably, John is trying to expose everyone to an array of models, (see Munger on models, short intro: http://www.psyfitec.com/2011/09/mental-models-arrayed-on-charlie.html ) If it were easy we would all be rich. People disagree, there are different types of value investing, but at the root as Munger has said, you are just looking for a mispriced bet.
    So, as to mechanical investing. Of course it works. Irrefutably. How do I know? Well that is the benchmark against which we are all judged, the indexes. That is what an index is a mechanical selection and 80% of institutional investors can not match it. This perforce proves the worth of mechanical investing. Were life “fair” and without trading friction, well maybe half would be better and half would below. Well guess what, life is not fair; there is fiction which kills that notion. (Statisticians, hush up, I’m just making a point here.)

    This is not a trivial point that the S&P index is mechanical. Nor is it trivial that equal weighted index generally beats the cap weighted index standard S&P 500. At the heights of the efficient market theory’s reign in academia, the value (and size) anomaly was known, but greater returns were allegedly due to increased risk, but the anomaly had nothing to do with risk. And, trust me, the efficient market academics used all your arguments and then some to explain away returns from mechanically chosen value names.

    On a technical note, as some commentators have said Greenblatt and others have corrected for survivorship bias, etc. These are or were known problems.

    Dr. K made some other errors, also that others have pointed out: that Graham was not mechanically inclined; well, buying netnets were absolutely a mechanical investing method. Also the citation of Graham, referred to market prediction….But somehow there seems to be a deeper issue here, and pardon the speculation, but it seems that there is an almost romantic notion that people should be able to beat machines, i.e. algorithms. Some can, but as the indexes show, most can’t. As pointed out in Quantitative Value, in field after field the machines (read the algorithms) are winning. Why you might ask; well this is why at least in finance: Very few people could quietly sit with a portfolio of Berkshire Hathaway, Intel and WalMart and not trade a thing for 25 years? Hardly a soul. Despite the fact that you would be securely rich, if you had. Despite a compounding growth of greater than 20%. (And your broker would have starved to death.)
    That is the problem.

    (Please note that the pasts of Berkshire, Intel and WalMart definitely do not predict the future.)

  14. OK. I’ll respond to a few of you at a time so I don’t get confused.

    To tjhanch. No my dog get not get kicked by a Net Net stock. One of them died last summer of a heart attack. The other one is 13 and still alive. I highly doubt he cares about deep value investing!

    To Canadian. I’m glad that you will find the Special Situation resources very useful!! Perhaps we’ll do some Special Situation analysis study together ourselves?

    To Charlie Munger. Well I did watch some videos of Toby Carlisle and he did indicate that you won’t perform as well if the investor “over rides” the screens he mentions. Most proponents of mechanical screens do give the impression that you should trade every stock the screen outputs and that’s the problem!! I did not miss the point. Screening is not valuation!!

  15. Chad; No I did not apply for a job with Joel Greenblatt or Toby Carlisle.

    When I wrote these emails to John I did not anticipate that he would copy and paste them directly to his site. I thought he would spend more time reading them and then rewording them to make them sound more friendly on his blog. I don’t regret anything I said however.

    Indeed I do have a lot of knowledge about investing. I am not trashing John and John most certainly does not feel that way!. I am trying to inject very constructive criticism into this course to make it much better. In fact when I did talk with John over the phone he suggested that I head up a group within this group that wants to study at a more advanced level. He offered to help me in starting the blog! I think I might very shortly take him up on that offer!

    I would love to start my own RIA firm and manage money but it takes start up money to do that which I don’t have! You ask at the end of your comment where are my books and my suggested methods. Well I gave a lot of resources about what books I would suggest.

    No I have not read Deep Value in it’s entirety. I have only read to the end of Ch.3 but others have and so far it’s read like a “mechanical investing” book to me.

    My attitude is just fine, Chad. I have seen a lot of Wall Street investment fads come and go and seen many people get destroyed by bad investment advice. Sorry if I gave a “pissed off vibe” but instead I should have given a very cynical and skeptical vibe. That’s all I’ll say in response to you, Chad. I am not going to exchange insults with you.

  16. thparadox; Perhaps the answers to your questions are in some of the papers I attached above.

    You asked if I am directly criticizing Greenblatt/Carlisle. Well here is a direct question. David Baily mentioned about knowing or not knowing how many trials (or N count) was used to arrive at the given results. A very high N count is called “Survival of the fittest.” Eric Falkenstein discusses this in his wonderful book. How do we know how many “N trials” Toby used before he arrived at the results that he published in his book? I think Bailey’s point is that their is no reliable way to know what the given reseacher’s “N count” was!!

    I guess I was not directly criticizing these two but the entire concept of back testing in general. The Campbell Harvey, David H. Baily and Numeraire papers are a little heavy and require some deep thought. Indeed I don’t 100% understand everything in those papers but I do understand enough to be very skeptical of mechanical back testing.

    I think mechanical investing will not be a fruitful path for those who choose it. You will be disappointed with your results.

  17. nrh; I am fairly humble. In fact one of my motivations for writing all of this is that I want to get more practice and feedback from selected qualified people. We are all guilty of Jer. 17:9. I am no exception.

    Perhaps we both can moderate this new advanced start up blog or subgroup? Should I have John give you my personal email address is discuss this further in private?

    Canadian do you want me to contact you?

  18. Pingback: Bamboo Innovator Daily Insight: 12 Feb (Thurs) – Jeff Bezos’ best piece of advice to entrepreneurs: Be missionaries, not mercenaries; Rethink Your After-Work Routine to Make the Transition Home a Happy One; Fatigue and Stress Fuel the Tendency to

  19. Another great book that highlights the folly of mechanical investing is an old book: Winner Take All by William R. Gallacher. I wish their were many more books like this one instead how many investment books are written now a days!

  20. Thanks for the links on backtesting etc. I am a quantitative value investing ‘fan’ so it is a good mental test to read some opposing thoughts etc

    I’ve only just started looking through the links but have yet to see any smoking gun that shoots down Carlisles and Greenblatts conclusions.

    For example doesnt HML&MOM both pass the more stringent t test criteria in this paper?

  21. Just read the article ‘What is circular reasoning’. Describes what it is but how does it prove mechanical formulas are “garbage”?

  22. Hi Guys. I just wanted to say a couple of things. First to John. Thank you so much again for this site and the course. It is a wonderful thing you are doing. I realize how much work this is. To do it for free and to be so open and engaging is just wonderful. Second, to Dr K, thank you also for the amount of time and effort you put into asserting your argument on the pitfalls of quantitative or pure mechanical approaches. I think your points are completely valid and extremely valuable.

    I may be speaking out of place here, But I imagine part of why John decided to set up this course and site was to teach, and by doing so to learn much more deeply and also to get extensive and critical feedback. Its the quickest way to learn. You have pressure on standards (don’t want to get embarrased) and also direct feedback. Furthermore, many can benefit in the process. Its an incredibly hard thing to do that. I dont have the stomach for it.

    For Dr K, to spend so much time addressing a valid point of concern is also incredibly hard work. I didnt read what he wrote and get the impression it was an attack at all. The point on being aware of biases in research (whether they apply to the authors mentioned or not) is completely valid. Moreover, I am struck both by what Greenblatt said in his lecture at Columbia (about data accuracy – and the amount spent on infrastructure to make sure data was clean) and then in the same lecture series, Glen Greenberg made a pointed reference to the same thing. How he had moved away from Capital IQ and other such sources given inconsistincies in data. Not that this is completely endemic, but it can affect results both in giving false positives (which is not so bad as you will likely check the output manually yourself if prudent) and also more harmfully in false negatives.

    I have seen this in my own experience. A lot does get missed, or misreported. Its a problem for the lay investor. To Dr K’s point, and I think its very fair, screening is not valuation. Just low EV/EBIT does not constitute value. Their can be data error, so the multiple isnt real, or there can be off balance sheet issues, and a whole host of other mitigants to Value. Their really are pitfalls in being purely mechanical. That seems true. But I dont think at any point John ever suggested that alone was a good idea. Nor did I read into Dr K’s comments that just because of these flaws, that these screens dont serve a purpose. He is just distinguishing their purpose, a filtration device to make the search process narrower and more likely of success, from the idea they are in and off themselves fullproof. Anyhow – that is just my 2 cents.

    Beyond that, from my own experience in some investing fields that are very data and quant heavy. Even in those fields, with huge dedicated resources, data issues and reporting are terrible issues. It really does lead to error. Beyond that, being purely quantitative can give a false degree of confidence. I think its less of an issue in simple screen measures like EV/EBIT – and one can do after the fact research relatively quickly to get rid of false positives. But in my experience more money has been lost by holding onto faith in a model with a subtle assumption problem, or that has been subject to overfitting. More likely the more factors in the model. Its ironic and sad that the more faith one has in the model the more likely one is to get hurt badly. I can elaborate if anyone has any interest.

    I just want to say again, Thanks to John for creating this learning space. I love it. Thanks to Dr K for doing what was asked and implied, and giving critical feedback and pointing out the pitfalls. And thanks to everyone else for pitching in. Hope everyone is well.

  23. Valuefactors; Well I can tell you that I did call Campbell Harvey and tried to pry out of him the authors and risk factors he felt meets his adjusted t-stat threshold and which authors and risk factors did not meet his improved adjusted t-stat threshold. He declined to comment. I can see why. Look at the controversial reaction I have gotten when I have suggested that back testing published results are mostly false!

    In the Robert Coleman/Numeraire papers he is saying that many/most of these various factor models have logical errors in their construction. He cites CAPM and French-Fama models as examples where the left hand equation variable (Return) is repeated on the right hand side of the equation.

    I guess what I am saying is that these authors are not going to come out and directly say most of the time “such and such of author is false or is a scam.” It is up to you to determine of if a particular author or model violates what they are trying to tell you. For example understand how Robert Coleman wrote the Return equation (with the visual detection color test) and how he wrote the right hand equation. Now what you could do is look at the Campbell Harvey paper where he list 500 some factors etc. and write these equations for every factor the way Coleman did. Then you can “visually see” what factors can be combined to be completely independent and which will create economic simultaneity.

    These papers are advanced and it’s understandable if you can not see the errors these authors mention. I am not 100% positive that Greenblatt and Carlisle violate these conditions but it’s extremely probable that they do. Quite frankly after reading these papers (and understanding them) the burden of proof is on the author/promoters and supporters of mechanical systems to show that they don’t violate these conditions.

    Someone (I can’t remember who) suggested or asked that so far they “had not seen a smoking gun” that refutes the logic of Carlisle and Greenblatt. The best response to this is to read the David Baily paper. It’s clear to me that Greenblatt/Carlisle developed their models using regression. Bailey makes it clear that most (including these two) are not aware or do not disclose the N# of trials or “failed attempts” until they arrive at their final equation and conclusions. That does not mean they are dishonest but they are not aware they have created a over fitted back tested system. I think until the financial industry figures out a way that an author can “prove” how many N# trials he used to arrive at his final system then 99.9%+ of most published papers and mechanical systems should be viewed with skepticism.

    I am not surprised that many people so far have been defensive and have “killed the messenger.” History is filled with examples of man being defensive when he has spoken out against “established knowledge or authority.” I can understand why some of you felt that I was too harsh in my initial posted emails but perhaps over time when you acquire the experience I have and read and study carefully the suggested authors then your perception of me will eventually change. I went back and re-read some of my suggested papers last night. It’s up to you to respond to them gradually over time. I can imagine possibilities where you will be able to overcome quite a few of the objections I mentioned. However I fell that the David Bailey papers about the N# of trials is the most compelling because even if you figure out a way to comply to the other auhor’s objections I don’t see any way at this time that you can “prove” what number of N# trials you used and that you did not “over use your Bailey N# trials quota.”

  24. Shaun; Thanks for the kind words. You are understanding me correctly.

    About what impression John gave about mechanical investing. Many have “corrected me” about who has and who has not been a proponent of mechanical investing. Quite frankly I don’t care because my objections to mechanical investing are the most important points you need to grasp not who said or did not say what.

  25. Two other good books that highlight the folly of mechanical investing wisdom are:

    1. Why the Best-Laid Investment Plans Usually Go Wrong by Harry Browne. Read Ch.1-16 but especially read Ch.11 and Ch.15. I don’t agree with what he said in the 2nd half of the book.

    2. How I Trade for a Living by Gary Smith. Gary is a momentum trader and not a value investor so I don’t agree with his investment approach but Gary is a staunch critic of a 100% mechanical approach to trading or investing.

  26. Here is a quick list summary of my objections to mechanical back testing.

    1. Economic simultaneity & Logical errors in the model’s equation. See Robert Coleman and Numeraire papers & articles.
    2. Publication bias in the financial field. See Campbell Harvey papers.
    3. Author of the mechanical system does not disclose the N# of failed trials before arriving at final system equation. “Survival of the Fittest”. Furthermore at this present time their is no reliable way for the author to prove his N# of trials. “Take my word for it.”
    4. “The Black Swan” or “Peso Problem”. See papers about “the Peso problem” and Nassim Taleb books and papers.
    5. Survivorship bias and de-listing bias in the database or dataset the author used when designing his back tested system. See Eric Falkenstein’s book, papers, articles and videos for more about this.
    6. Measurement errors when trying to quantify the system’s Alpha. For example when comparing the system to a given benchmark the Alpha may be very sensitive to the benchmark you are comparing it too.
    7. No off balance sheet accounting in the dataset or database the author used when designing his back tested system. This is why most screening systems are almost useless. See New Constructs for more about this.
    8.The Alpha measured by the author of the system may be too sensitive to the rebalancing scheme he used. Equally weighted portfolio vs. other weighted schemes.
    9. Most papers are system designers have a hard time estimating the true cost of executing the system. Most papers don’t include Transaction costs.
    10. The Geometric return difference vs. the Arithmetic return difference. See Eric Falkenstein for more about this.
    11. Problems replicating the system or screen. For example a system might claim an Alpha of 2% over a benchmark but to form a diversified portfolio you must own 50 stocks equal weighted. To do this would cost a lot of money so only a very large investor or fund could do this. Smaller investors would have to “cherry pick one or two companies at a time” and would have to form a broad based portfolio over a longer period of time.

  27. I take Dr. K’s point, but only half-way.

    When does mechanical investing cease to be mechanical anyway? People make investment decisions based on their own set of heuristics. Mechanical investing turns heuristics into crisp rules. Heuristics may or may not be difficult to turn into rules. For example, “low PE” is easy to translate. “Quality” is more ephemeral, but even here, people have devised measures of quality.

    So, if mechanical schemes fail, they fail because they were either conceived erroneously, or because they failed to capture a heuristic used by a superior investor.

    But that doesn’t mean that mechanical schemes are bound to fail. There is, after all, accumulating evidence that they can outperform human experts.

  28. Hi Everyone. Me again. This may not be useful. And I think its a variant on one point Dr K makes on the peso problem. This is from my own experience. But I think it relates. That said it is completely unrelated to company valuation. But it does speak to mechanical investing problems.

    There is probably no area of finance so data heavy and data mined as the mortgage backed space. Be it Agency MBS or non-agency. Every major participant in the space has enormous quant resources, and has had the past 20 – 30 years. Its an enormous market, and enormously complicated in parts. It has historically attracted the “brightest” in finance for periods. Given the enormous resources put into both data mining and modelling, it is probably a good proxy for this discussion. There are reams of studies available on the various model factor impacts historically, their reliability etc (factors that affect prepayment, such as interest rates, divorce, etc).

    one would expect that the best in the space allow for as many contingencies as possible. And given they use monte carlo simulations en-masse they would have you believe they do. Yet, repeatedly people have blown up in the space. In 1994, 1998, 2003, 2008/2009. Usually for different reasons than the prior experience. But blown up nonetheless. The ones who havent (who are also extremely quantitative, and I know many of the people involved reasonably) are those who see the exercise as not purely mechanical. The best I know described it as an art (which you wont hear him say much in public). Sounds a lot like buffett, no? Now why wont he say it in public? Most of the investors in this space dont want to hear that. They want it to be math and objecively driven. It gives them comfort. Thats very appealing.

    Incidentally, the reasons for blow-up are instructive. In 2003, it was rate related (Honestly, they spend hundreds of millions modelling this), and the effect on prepays. In its simplest senss, as rates move lower, the borrower option to prepay goes further in the money. So prepays (CPR) should pick up. This was modelled. There was a huge correlation across models used by various people. Fairly based on existing data. People just werent prepared for the extent of the rate move. Models (I forget the exact numbers) projected CPR to jump from say 15% to 45%. And people traded based on this. In the hundreds of billions. Over the next 2-3 months, as the realized CPR’s were published, the numbers peaked @ 30% CPR. This was an unmitigated disaster. Wiping out many years of prior profits in a number of places. Why? It was a logistical impact. Rates had been lacking vol for a while. There literally were not enough people @ the various banks to process the volume of refinancing. The models projections were logical, but made no allowance for this bottleneck risk. Most all people got it wrong. Logistics of the process wasnt something they considered.

    You all know 2008 and the risk of the guarantee being called into question. Even beyond that, in non-agency space, people had modelled the reaction to high CDR (default rates) but again made a logistical error. What would be the impact of this huge wave of defaults be on the process of foreclosing? Particularly in judicial states? Timelines extended over a year longer or more in many cases in many states, which has a huge impact on the PV of what you get with your bond, as well as expected recovery. Again, thinking to the second step, how things might work in this state of the world (rather than extrapolating the number you have witnessed linearly) makes sense. Yet I know nobody who did it (myself included).

    Mechanical systems, even in places with reams of data that has been tortured to death cannot be relied on blindly in my experience. Even in Deep value, or Quant value, it is interesting to see with long stretches of data how much performance of different measures, (like P/Book, or P/E) vary over different holding periods or sub test periods. I think those principles work in general, dont get me wrong. But I think one should pay attention to what DR K is saying. He is not trying to pi$$ anyone off. He is just giving the risks to such an approach. the implied additional return may be a sufficient mitigant for most. Its a personal decision. But wouldnt you rather hear the other side of the argument? I would love to.

    My own view based on all the above, is that I am extremely wary of data mined output and the risk of overfitting. Torturing data is like torturing people. It will give you any answer you want if you push hard enough. So as Einstein warned, be careful, the easiest person in the world to fool is yourself.

  29. Complicated , overfitted models have the danger of producing erroneous results. We all know that. So don’t build complicated overfitted mechanical models!


    Psychologist Gerd Gigerenzer studied the use of heuristics in decision-making. He found that simple heuristic models often outperformed multi-factor statically-fitted models. As noted, there was a danger of overfitting that the simple heuristic models didn’t do.

    I take your point that models may have hidden risk assumptions in them, making them vulnerable to black swan events. And it’s true, many models work until they don’t. Evaluating mechanical models is like evaluating investors. It’s difficult to “know” anything for sure.

  30. Hi Dr. K,

    Thanks for the comments.

    “Burn Rate” was actually first discussed by Ben Graham in his book Security Analysis. If anything, I added a name and attempted to explain the principle as I see it.


    While I only use my Net Net Hunter NCAV Scorecard and that ball of neurons between my ears, I have heard nothing but good things about Jae’s spreadsheets.

    All the best,

    Ps. Oh yeah — and the reason why Graham was against formulas was that if people found one that was successful then everyone would start to use it until it stopped working. With net nets, since the companies are so small that the pros can’t touch them, the investing public so transfixed on big growth firms such as Apple, and the distaste that most investors have when it comes to buying terrible companies, there’s a lot less risk that people will pile into the net net stock strategy. At least, that’s how it’s tended to workout over the last 85 years. 🙂

  31. Evan; I think you have done a much better job of advancing our understanding of Net Net investing then Toby did with his books. You are not 100% mechanical which is how real life successful value investors really behave.

    I think Jae Jun’s spreadsheet program is good but it does not do any off balance sheet accounting.

  32. Thanks, Dr. K. I hope I was able to add value to your investing.


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.