noSQL

Firstly, sorry for all the crappy blogs, just something I'm working through, each draft is so different from the next and eventually I just post it.

Anyway, it's funny how ones conscious attention works in computer engineering.  Case in point, I have been exposed to NoSQL and a lot of the tools (by name, not by use) but it didn't click with me at all.  Now, writing to a linus virtual server cloud, it had occurred to me that not only did I not want to insert an RDBMS from the start.  I think they put constraints that impede iterative development, but they are also very useful and powerful... so the idea is to bring them in exactly where you need them, hopefully late.  So I  started working on and designing a different kind of database approach.

Turns out of course this is the whole point of NoSQL, and they put it the advantage as "schemeless".  These components are easy to work with, so even if they are overkill... at least they oppose almost nothing on the internal design. The decision I made to communicate via JSON/HTTP is also present in some of these tools, as is the general idea of distributed associative arrays.

I have found, after a closer look at MongoDB (something I'd heard the name of quite a while ago but not looked into) in reaction to a youtube comment in a video on the subject, that Mongo will do.  Now after another day or two of looking at it I'm very happy with it... it performs well (according to benchmarks) and is easy to link to in C++ and Python, and has a decent shell, and it seems, a lot of other interfaces and tools.  Mongo basically stores and retrieves Nested Associatives Arrays by namespace and searching.  Indexes optimize this and the storage is BSON, which is a binary nested associative array.  This is what the JCN does, but I had no storage, thus this tool search.

What that leave me with now, as far as responsibilities for the JCN comes back to what it does for the video system.  The python web processes upload a video, writing both the video data and the metadata (in a JSON file).  The JCN sees this (using inotify to watch the ingestion directories).  Use of mongo means I don't have to have the JCN return arbitrary JSON documents (things like user settings, etc), and instead it will focus on integration of the data/document store and the file system.

For example, to finish porting my Novem9 interface system off GAE to this LVSC (linux virtual server cloud, aka OpenStack), I need to be able to store "articles"... the equivalent of a blog post or what have you.  As you can see here (http://www.novem9.org/n9/pds).  NOTE: I'm not pointing there because of content, the content doesn't matter and is obsolete... but for the interface system.  Those articles can be edited in place by a sufficiently privileged user.

I need to do this on the LVSC. But also there is a feature I didn't put in GAE yet, that I would like to have, which is revision control for the article as it is edited.  With this MongoDB and JurisdictionalAgent(my C++ daemon) the latter will allowed saving the contents of the article on disk (MongoDB just has the path)... and the JCN can support commiting changes, looking up old versions and the like, while MongoDB ensures the article can be found in searches.

I feel back on track.

Jurisdictional Nodes: Role of Relational Databases

Firstly let me start by affirming the great power and overall quality of relational databases. All praise be them.  I'm not being sarcastic, I just want to load up on the appreciation prior to possibly criticizing overuse of them. Or I might merely be fantasizing about a world where they were not so necessary in some circumstances, and it might even be in vain.  As I say later, I consider them worthy, reliable, and necessary tools  which excel at indexing data in a general way, probably crucial for arbitrary data mining especially for business research. There may be no other way to understand how your resources are expended and utilized as an operational entity.

RDBMSs are reliable repositories for millions of records at a time, they are generally well written, and they are very well researched in academic computer science and are generally constructed to use some of the most powerful indexing and searching techniques from computer science in general. The first relational database I ever worked with was Ingres in the 80s, since then I've worked directly with Sybase, mysql, postgres, sqlite and various ORMs (Object Relational Mappers). In addition I've worked indirectly with many more, from Oracle to IBM databases.  I don't expect to ever stop using them throughout my work. But this essay is about where I would prefer to use them, and an attempt to relegate that to that position. My current goal is to delay use of them chronologically, ideally to an end point where the verticle application structure does not depend on one, but all useful data can be exported to database tool to enable analysis with standard tools. At this point I would rely on a RDBMS for long, massive queries, but not for the small everyday queries needed to support individual users.

So not only is this not a hit piece and neither is it uninformed on the topic or practical utility of an RDBMS.  This essay is me exploring an idea that frankly has an escape hatch.  I can map what I'm doing to a relational db at any point, and the later I have to do that the more successful my experiment will have been. The less runtime dependence, the better.  As such, it is an open research, an in progress communication in the spirit of R&D, I seek, if anything, rational proof for and against my ideas to guide me.  I expect drawbacks which will have to be met head on to be mitigated or  ideally, to be solved.

Having said that I think I can afford to put the goal bluntly: I want to make a mission critical capable stack that does not rely on an RDBMS at runtime, but which will export data to one for use by standard issue tools.  Therefore I don't intend to duplicate all the abilities of a DB by any means, I merely mean to free the application from having to work with one on a constant basis for mundane tasks, like finding a small set of recently created objects, or objects associated with a particular user.

Here is an attempt to give a sketch of the sort of things that motivate me to want to do without an RDBMS, at least during development, and ideally minimized and only aimed at after-the-fact data mining.  Why I'd prefer not to let RDBMS concerns affect the entire design:

• RDBMSs cooperate poorly with iterative development.  Say I have an object type A for which I want to develop feature sets X, Y, and Z, in the long run.  I am inclined to develop these one at a time, of course.  Assuming I created a table to store information about objects of type A, it will incorporate first the needs of feature set X. Thus the table in an RDBMS will at first reflect those needed by X, then by X+Y, and then by X+Y+Z.  Since there are likely to be relationships between X, Y and Z, not only will properties/columns need to be added (something which was not supported when I started with relational databases), but I expect that along the way I will probably find ways to consolidate properties needed by each feature set.  That is I might create something for X that needs to be generalized to support Y instead of adding a separate column/property altogether, creating more abstract properties with more flexibility. The RDBMS is an incooperative dead weight when making these exchanges.  It's all quite clear why it's necessary, and indeed, it's amazing how flexible they have made them in modern times, such that columns can be added after the fact. To do this properly without clearing the DB and starting from scratch each time, is to create special software, or special plans, to survive each transition. I have worked on a variety of ways to help regularize such processes and I hope to see that such work can be lessened greatly. Forward compatibility issues are crucial for iteratively developed software.
• RDBMSs are general purpose. That is great, especially when needed, but the general danger of such tools are that they solve every job with average performance, when a special purpose tool solves a smaller domain of problems more efficiently. While the RDBMS speeds up a massive query they will never be any faster than a custom written search using appropriate indexing, but in many cases they will be on a par with that result. This means one of their strengths is that they save development effort thinking about the logical particulars of one's particular problem domain, and in general developer time is worth a lot of efficiency in the code which can, after all, be optimized later. But obviously the drawback is performance less than optimal, and the value of that difference is dependent on the task at hand. The RDBMS stays prepared to execute novel queries on any given property.  Fact is, however, you are never going to search for blog stored in an SQL DB given the text of the blog as search parameter. Some of this generality is overkill.  If job responsibility is partitioned well, shedding unnecessary generality is an opportunity for new mission critical systems.  The generality is regained over time, and one must estimate the likely that such an outcome is possible, and the approach doesn't have internal contradictions or bottlenecks that cannot be reasonably mitigated.
• Use of an RDBMS requires one to model data as objects made of exactly the same properties, distinguished only by different settings. This is never the an ideal model for the program consuming the data.  A program necessarily needs to have its own model and will, if at all well built, will communicating with the RDBMS by querying into and out of the model.  That means that I still have to develop the appropriate data structure model for my data as used in the program throughout the stack, while also creating a separate RDBMS-oriented model. I worry that efficiency concerns lead people to subsequently design entirely around the what one views as a required RDBMS model.
• The RDBMS hides the storage from us at the shell level.  During development, and even in deployment, I find it useful to have data access available to me at the file system level. While I respect the model incarnate in the RDBMS, I also appreciate the unix name-pipe/file system model as a very useful heterogeneous data abstraction, albeit one that is poorly indexed relying only on lookup namespaces which are generally invented before the data itself, in the reverse of the proper order. File system elements are easier to inspect and hacked from time to time. I can also use standard tools not designed to work with a Database.  There is a considerable amount of power in the unix shell, and none of it is of much use dealing with data stored in a locked, binary, cabinent that cannot be poked around with.  This means that solutions to problems that might have a quick remedy may be long in coming as a more formal fix is required.  A custom system can react to any peculiarity of the data or  circumstance, and then plan a path to a formal, steady state, answer.  This is the kind of flexibility I want in development and afterward.

 Situation

This is an experiment, but not an academic one, more of a standard R&D ambition, based on a hope that the benefits outweigh any drawbacks at least for the problem domain targeted. I'm using it in development right now so let me start by explaining a chronology of events that will explain where I am so far, and which ideas I hope can carry this to a successful end. The narrative should also reveal any errors in my thinking that I, or another, might thus find visible and rectifiable. I go in not with conviction but of exploration.

Over the last many years I have built various sets of software as projects on my own time, sometimes to explore new technologies or to tinker with long term ideas of mind.  Most valuable from my point of view are my alternate interface to youtube and notation app, a javascript graphical two dimensional tile piece and board game interface, and a modular client-server application model based on a geometrical graphical design orientation. Most lately I have built a video and image processing system designed for linux virtual servers. I am now converting all of the above to the linux virtual server marketed as the cloud. 

While the data "cloud" did not strike me as significant, the ability to cell CPU as a commodity does impress me, especially when the host is a standard OS rather than a proprietary system like Google App Engine.  Thus I am converting the prior code, which runs on GAE to ubuntu and apache based server code. The first program I converted was a script I have that allows one to specify six images with the subsequent result that the images are cut up and put in an single image such that if printed, the image can be folded into an origami box such that the six images appear on the six sides of the box.

I obviously did not need an RDBMS for this application, which I called photogami, which was nothing more than a stand alone script converted for running on a web server.  However, if I were to promote the site, if it were to become heavily used, such a need would naturally arrise even for such a simple application because I let people upload their images, for example, and I would in that case be well advised to keep track of who was using what.

Subsequent to this I developed a video server with the initial issue being mastering ffmpeg which is used to convert videos on the back end. I needed to have a login system so my system was not abused, and this is something one might likely address by keeping a user table in an RDBMS. But instead I used apache authentication, and put off use of a DB.

This was possible through to the point where the videos were successfully uploaded and converted (and for good measure the server could automatically upload them to youtube as well, if desired).  But at that point I hit the first stopping block.  The issues is this: while i can safely let the web server upload a video, and put it's meta-information in a file on disk for the first time, I cannot safely do subsequent manipulation of the data.  E.g. if I want to put that video id in an index of "recently uploaded" files, I cannot do this with the web server app because it is running multiple processes and those processes have to cooperate somehow to ensure they don't overwrite each others work which is what would happen if they simply write to files at will.  

It's crucial when modifying shared data, that something controls access to the official data and ensures a transactional and ordered nature to the modifications, which are required to keep the data complete and coherent.  Also, such a manager will cache the data so you often don't have to go back to disk at all.

Note, this is the least of what the DBs do, managing concurrent access to common data.  For that purpose alone they are certainly overkill, BUT if one is of a mind that they will have to  have a DB eventually to support their system, this is one of the first reasons to use one sooner rather than later. At this point I decided again to defer the need until later, and instead made a C++ component that would do this part of the work for me. I named the program a Jurisdictional Clerk or Jurisdictional Control Node. Data under its jurisdiction, somewhere in a file system or file-system-like data store could only be modified by it.  Theoretically readers (the web server processes) could read the data themselves, but must modify and write through the JCN.  In practice, they of course should (and do) also read the data using JCN to control I/O.

But this means I as a developer can inspect the data with shell commands while the system runs.  I cannot modify the data safely unless I am sure it is not being written, which I can ensure by making the JCN react to a command which locks that file from writing.  Or, by brute force, I can simply stop the JCN, ensuring no one can write to it, allowing me to perform developmental surgery on the data as need be, perhaps running scripts to handle a transition, as above, from feature set X to feature set X+Y.

This is relatively straight forward. While we often use database to look things up by name, it's not needed for that.  A file system is already such a database, indexed by path specifiers which allow quick access to arbitrary data. When they ask for the data from the jcn, it is read only once, when first requested, and subsequently one gets the data from jcn memory (this takes a thousandth of a second for the data structures the current size, which is similar to a row of information).  In this case the path is LITERALLY the key (in C++ map<string, Morsel *>), where Morsel is a class I have created which uses boost property trees to keep heterogeneous trees of data, and can write them to JSON, XML or a native format.  

Which format to use is thus generalized, and can be changed throughout development in reaction to differences in performance (including to an RDBMS, aka the escape hatch).  Because I'm maximizing access to the data during development, for inspection and manual manipulation, I'm using JSON at the moment.  This is also useful because this format means I can send data unparsed to the client, instead of doing a query and formatting it as JSON for the client, since the client is a javascript program which of course uses JSON natively. Naturally, I suspect for sites with a lot of traffic, a text format for the disk storage is not going to cut it, and the Morsel is the interface which allows us to put data in less convienient but more efficient binary formats by changing only the Morsel class.

A nice thing about the JCN approach so far, however, is that even when that optimization becomes the case for  a particular object class, it needn't apply to them all.  While object classes with a lot of particular instances, like comments to videos for example, may need a more efficient format or storage system, something like the site configuration file needn't. The JCN and its service classes can use the less convienient but more efficient system selectively, for things that call for it only, and also only at that time during development where it's really become necessary. Indeed, it's possible that throughout development everything can be kept in the easier format, and changing this in deployment can be merely a matter of changing a configuration flag in the object class definition files.

So the JCN started as a central point of contact for the video bank software data, especially the meta-data, such as the description, uploader, view counts, and so on.  Rather than have the web server scripts contact the jcn after a video had been uploaded, I instead made the JCN use inotify to watch files appear within its jurisdiction, which it can then react to.  Initially these were specific to the video server needs but I have since, as part of converting the Novem interface system to this new server, started to generalize the JCN and give the design more definition.  The morsel is a data abstraction which contains a nested map (or "dictionary") of property:value pairs (that is, values can themselves be maps) and which borrows the concept from javascript that such a thing actually IS an object.

While the JCN currently performs its video services by watching for files of particular extension to ingest (let's say it's .vidinfo), the more general solution that will come from the recent modifications will be that if any file of type ".mrsl" appears on the disk, the jcn will ingest it and expect to find type information in the .mrsl. The dictionary can (and to be handled by the JCN directly, must) have an "_obj" member, which contains properties generally intended to be known by the jcn or other control system, meta-data such as the object's current namespace, the objects name, its object classification, and so on.

A video information note would have the objectClass member of _obj set to "videonote", allowing the jcn to dispatch the data to specific handlers that actually know what to do with such a thing (e.g. that know what the date, author, or other fields might actually mean). I have another class to act as the collection, the database if you will, of these Morsel instances, the Prandium.  This class is what keeps the map to all the morsels in memory.

I feel confident with this design. Right now I can let the prandium keep track of data that is in principle on disk, and can cache it in memory to save IO, confident that I can make the store a more well indexed file if needed, or from here execute the escape hatch and link the Prandium to a RDBMS and at least keep the prandium's interface, which is designed to my convienience. 

 This all works quite well and I can see the implementation will satisfy the requirement mentioned above and will act as a competent (efficient) shared datastore manager for retrieving/modifying data by "name and namespace" (i.e. the "object path").  My current system support a path language within the data morsel as well, so the JCN can allow arbitrary modification of these type of structures (heterogenous trees).

I also said above that while an RDBMS is often used to do this sort of lookup that such a job is not its real strength. The real strength of the RDBMS is its ability to do queries efficiently, especially important for very large queries.  The ability to do queries is something I need right now.  The JCN will currently provide a list of current videos, but it will not cull them for you, you can't ask it for the latest dozen videos, for example.

The RDBMS excels at doing such things, and is ready to do them from the moment you put the data in.  Generally, any property of a row (aka "column") can be used in a query. Thus, especially with the wanton aid of ORMs (which is indulgent to the OO programmer), one creates a column/property for each property one desired to have stored in the database. This means you can query on any of these in the future, without needing to have forethought about what you will search on. 

But the fact is, many of the properties in the database won't ever be used in a query.  The text of a blog entry will not be used as a query for the blog object in the database.  It has occured to me that for business reasons it's valuable to be able to search on any given property, because this supports arbitrary business analysis, and for this reason I do expect to export pandium and morsel data to RDBMS for such studies.  I am not trying to remove the RDBMS from the whole endeavor, I'm just trying to keep my server system from having one as an operational dependency for the runtime system.

A flexibility of the JCN store (again, remembering it can be put in various types of storage on disk and is already more or less ideally efficient when in memory, indexed by path) is that it has no column concept and thus you can put any data in these dictionaries. You can easily add extra sub-objects, which will be well ignored by systems that don't know where to look for them, but which can nevertheless be handled by the infrastructure by iterating over all the present keys, and also through the special _obj member.

However, this makes it difficult to use these things as arbitrary search parameters, and by difficult I mean impossible. While most of properties do not need to be used as search parameters and never will be, what about those that do need to be used for searching? 

The current situation: I support the storing of "blobs" in the jcn by pathnames.  The jcn can inspect objects, but currently only does so to find control information in the _obj member.  It will check the type and creator for authorization purposes, but in every other sense does not have to know what the type actually is (just that it's allowed). 

What I am developing next is in order to use internal properties as search terms. The idea that I will implement is that one can do the jcn equivalent of "create table" AFTER a bunch of such object have been stored.  However, the command will not be creating tables, but rather storing object class definitions which drive creation of indexes. For those familiar with RDBMSs, this create table action, which is done to tell the database that you have a type of data you would like it to accept and the format of the data, specifies both what can and can not be used as a search parameter, and what's stored.  In the JCN model, what can be stored is more or less open and does not have to be pre-specified.  For binary types that cannot be stored in the actual meta-information file, a field can simply point to a file (or URL), accomplishing much the same thing in effect.  Thus anything that can be stored in the filesystem can be stored (in a file system or an optimized stored able to retrieve arbitrary data by namespace and path).  Any file system level utility, like revision control, can also easily be used.

The definition of things that can be used as search parameters.  For each of these an index will be created and when creating and modifying that type of object, these indexes will be maintained and stored efficiently on disk.  This index is a simple map of the given property as key, to the given Morsels with that value.  The properties are given as path names which "path" through the property tree in memory.

In the long run I could be swamped by a need for indeces on all fields, but I know that at least in development I will have only a gradually growing and limited number of things I need to be able to search and filter on.  As long as the indices can get me to small sets (hundreds or thousands at most), I can also support final filtering on arbitrary values in the property tree for which there are no indexes. This means that if I get through development without an RDBMS, but find my system cannot perform in deployment (perhaps it never will, perhaps it's just not ready), that if I take the escape hatch at this future point not only can I still benefit from the  RDBMS (by pointing key parts of the storage model to an RDBMS instead of a file system, and changing the "create object class" flow so that it has to precede ingestion of data), but I can also continue to develop to my general model, a server capable of replacing the RDBMS in development and possibly production use, and using the RDBMS for what it truly excels at, REALLY BIG LONG MASSIVE queries, at which it is a rock star.

One advantage is that at this point it is much more fun to work on the jcn, and seems to require less effort than making yet another RDBMS data model in parallel with the datamodel that actually gets the work of the applications done. I suppose I will go straight for high performance in storage of the indexes, and store them in some very efficient binary format. Perhaps I hope to redeem the filesystem as the ultimate data store... one I have often wandered in.  It could be, I'm like that sometimes.

 

Moral Foundation for Wandering Atheists

Greetings former and soon to be former theists.  Congratulations. Now, a bit of an issue for you... your theist guides will have warned you that atheism has no moral system, and indeed, perhaps no epistemological system (no way to say what's right or proper, no way to say what's true).

Frankly, this is true of "atheism". Atheism has nothing to it but a lack of theist rhetoric. But anyone that might be called an atheist, as reward for avoiding theistic rhetoric, will in fact have some form of morality, explicit or de facto.  Hopefully you will already realize that individual atheists, generally speaking, do have moral system. Again, often this will be explicit, but will at least be some de facto system of morality.  Similarly for epistemologies, the patterns or reasoning they use about what is really knowable will exist since they will claim to know, or at least act as if they know, certain things more in comparison to others they know less about.

The trend for people that are newly atheists or are soon to be atheists seems to be to try rebuild all the intuitions they retain from theistic morality into a system exactly the same in as many respects as possible. This phenomenon seems amply recorded in the philosophy of the nineteenth century and earlier as non-theistic reasoning took hold.

I think no one that as a theist was against murder, ostensibly because of the fifth Commandment (possibly the 6'th) in that case, because, upon deconversion from theism, suddenly became an advocate of murder.  Indeed, when making a moral system, which is what you have to do in a non-metaphysical world, dear larval atheist, one needs a mean of judging their attempt at a moral system.  One way to judge your system is by spot checking it against deal breakers... such as my need to have a morality that explains why, when, or how "murder is wrong" or "one should not commit murder".

If I am judging a moral system, if it advocates or cannot rule against spot checks such as murder, rape, stealing, then I will judge it poorly. A good moral system should postdict robust moral facts like these, and then, being consistent among known or test cases, might be considered more reliable when used to assess cases closer to the borderline, things we wonder about morally.

As a philosophical person, having gone through a lot of introspection in order to reach your non-theist or soon to be non-theist state, hopefully you will be wary, however, of taking with you too much of your previous morality, and I would suggest it might be good to minimize what you take. Further, you will use logic and rules of consistency to not take anything with you that you can't ultimately justify.  That is, you will face the possibility that you cannot re-justify even these robust test cases. Don't worry, you can do that, but do worry, it's not necessarily easy.

The problem is, however, while the theistic explanations have evaporated for you, probably more than half of the conclusions and doctinaire dictums linger on within you. As a skeptic and relativist it is important for you to have a rational and reasonable moral system lest you surely fall back into a dogmatic one, perhaps nothing but a little more self serving.

Rather than attempting to follow priests not of like mind, you may try to take your lead from scientists. This will be complicated to utter impossibility for not only are scientists on every side of every issue, but also, they say essentially nothing about moral issues. Still there is much to misinterpret when taking a lead from science.

As an anti-dogmatist as well as a mere non-theist, it's crucial from my point of view that a moral system not violate any skeptical or relativistic principles of reason, and the system I will describe in brief does not.  So for example we will not expect it possible to create just ONE moral system for all contexts. Also, justifications for a morality will be logical, meaning, if you accept certain principles, then the morality can be argued for.  E.g. while it's not possible to non-dogmatically argue that stealing is absolutely wrong, it is possible to argue that if you do not want to be stolen from, then you ought not steal from others, given that you want people to treat you consistently, and thus should treat them consistently.

(1) a moral system is a system that an individual uses to advise themselves how they should treat members of groups in which they belong
(2) even in an ideal situation, an individual will tend to have at least one distinct morality for every group to which they belong
(3) a moral theorist will naturally, over time, find common forms for moralities, common and distinguishing features, but given many moral theorists, and the scale of the task, even if it turns out possible to state a universal morality, which I doubt, one will always expect multiple answers from multiple moral systems in any single situation

Thus, lets say it is immoral to steal in all of a particular moral theorists known moral systems, and suppose as well that it may also be immoral to starve your children.  One can expect conflict when considering your obligations to your children in the near term, and considering stealing a nights meal.  In my calculation of this, the act does not become morally right, nor is it only wrong.  I hold that this is a superposition, that with respect to one group, it could be moral and necessary, and in another, immoral.  Such considerations inform a meta-morality which can deal with such conflicts, though the question is not particularly complex... a state with a police force will enforce the concept of ought and ought-not relative to the group of the state, but if well informed the state can apply a meta-ethic.  One can see that our system does attempt such analysis, and such considerations are not held to mitigate the wrongness of the act, say of thievery, but merely to advise as to the leniency of an appropriate reaction.

My moral calculus makes a clear distinction of terms:

(1) What is or isn't moral is an instruction for myself only.
(2) When someone else breaks my morality, the question is not how or if they ought to agree with me, the moral question is still for me alone, "How should I treat a person that aggrieves me?".

One can expect such a conflict, that is, someone violating my moral system, is to be decided by choosing from which group to make the moral calculation, and by taking into practical matters such as the degree of the offense and principles of respect for individual rights such that minor emotive grievances are not pursued at all, and only matters of material consequence rise to the level required in order to take action on transgression against persons which can be judged by a prior, agreed on, set of moral systems.

On the personal level, people can be expected to revert to the morality of whichever is their more close knit group. The ramification of this system is that moral arguments always draw from the shared reality of the group, all humans need food and water to survive, all living creatures need non-toxic habitats to survive, and so on.  Logical arguments can be made from acceptance of the group condition into moral advice.  This means we have to have a vibrant network of conversations, such that we know well what our shareable reality is, in order to make sound moral decisions as a group.

History of the US

I do not consider myself very well informed about history or especially the details of history, the many actual stories.  This probably makes it a good topic to discuss in that I am likely to be more humble than I am discussing epistemology, about which it seems people are in general deliberately clueless.  I cannot even get people to define a term like "God" before taking much time debating it.

Nevertheless I think I have some perspective on history which are well justified but which even those with a great command of the detail fail to see.  For example, people informed about history often blame ignorance of history on the individuals... after all, they have discovered history, why do their fellow citizen (obviously I'm talking about Americans here, but there is some generality to this in the west. It's just that in the Unisted States of America even intellectuals are only dimly aware of "history").

For example, something I have managed to discover is that even history still teaches us in the US the same false stories we were told at the time, and stories not able to tell this way, like the story of US intervention in the Phillipines.  We were told the hole time, and still are, about the US isolationism.  The fact is, the people may have been isolationist, but the leaders were not.  The Revolutionary was became an indian war, and subsequent to that America and her standing army recreationed in the Carrabean and the Pacific.

American have believed, and believe still, what they were always told at the time, in spite of history intervening with stark revelations which tell a different story, still they believe.

I think I struggle with this irony... on the one hand I want to summarize that small subset of my views which could be systematized to characterize my approach, while on the other hand this is not what characterizes one approach.  What more characterizes my approach is a description of my voyages and the tools I have found and used on them.

Logic

My more ambitious blog posts are in shambles, so let me start slowly.  Let me start with "logic". Specifically, let me start with an aspect of "logic" which is not widely or well enough understood for my liking.  It seems that the bulk of the curve, even those that regularly partake in discourse for their amusement and edification, do not seem to understand some simple characteristics of logic which are crucial if any real use it to be made of it in the determination of the affairs of mankind.

Perhaps the problem is that it is that logic lacks characteristics people attribute to it. Perhaps it would be easier to add a property to logic and than to ask people to understand its general meaning. People often thing that a "logical" argument has a conclusion.  In reality a logical argument says that IF some condition is met, THEN the conclusion is assured... IF some conditions are met. Or perhaps the problem is that a skeptical description of logic, while widely in use in science and engineering, is still not what is taught as the framework of logical analysis in the mainstream, so that even those educated in logic in mainstream universities will find it difficult to understand any very apt analysis of logic from a skeptical  point of view.  This is sad if true for this would preclude disscussing logic from  the level of abstraction necessary to understand the history of logic and the state of mankind's best understanding of rules of inference in general.

Put plainly, a "logic" is a system of inference.  The study of "logic" is the study of rules of inference. That there might be just one set of rules is clearly not an option, that is demonstrated. Study of various approaches to inference show sets of rules which cohere into separate, albeit related, systems of "logic". The rules of first order logic and second order logic are distinct and yet also bear a relationship. So too is it with modal logic, or fuzzy logic, or "logic" with any sort of qualifier meant to specify as least partially distinct characteristics.

At large with the so called natural languages of mankind one finds in use rules of inference which follow identifiable patterns, with some consistencies and some inconsistencies.  This holds true, it would seem, however lacking in formalization or however non-analytic the particular rules of inference may be, given enough people reason by said system of inferences.  Many such "logics" belong to categories of inference wholly held as fallacious by philosophers, which adopt sub-rules which logicians have tended to agree are unreliable and don't have a place in a proper "logic".  They may have gone overboard in such judgmentalism on the one hand, while on the other, it's clear they have a point considering how many bad rules of inference are still in use today which ought to be overcome.

This means that many rules of inference, which I would call a "logic" or "system of logic" are anathema to the traditions of philosophy.  These systems are anathema to them and generally we would call them illogical, which I can abide, and also prefer not to consider them as "logic" at all, which I cannot abide.  I understand some criteria upon a system of logic will over time be implied as a meta-logic develops by which one might measure and just a particular system of logic.  I understand calling some logics, illogic, but this can only refer to the incorporation of incoherency and irrationality, and cannot make us pretend that fallacious rules of inference are not inference at all.

An example of widely used systems of inference often considered illogic by philosophers are systems of inference which are purely heuristic systems, that is, they contain a set of arbitrary rules of thumb, learned by rote rather than by principle, as the latter are unknown.  A heuristic is not understood in principle, but is merely a matter of "if you see A happen, then B will happen".  These hueristics may be true, but not being understood, they are not very flexible, and tend to work for the bulk of cases and fail enigmatically as the edges of sets.  They are not themselves subject to the sort of reduction which allows exploration of subtle inferential details, a feature formal logic tends to intensify. An example of such a system of inference might be a persons ideas of what signifiers will reveal a person might be a good mate.  It may consist of lots of rules of thumb, this behavior implies that tendency, this physical characteristic or mannerism means that about their nature as lovers of home-makers. Given the context, where declarative statements cannot explain the conditions, first order logic is at a loss, and it may be only such crude systems can address the real human questions that afflict us in reality.

In a skeptical discussion of logic such systems are considered in addition to the formal systems which have the advantage of being technically defined and highly shareable.

Thus, the first thing to realize about "logic" is that it is not just one thing.  There is no one "logic".  When we say an argument is illogical we generally mean it apparently violates some of the known rules of first order logic, or possibly mathematical logic. While I myself am fond of making such criticisms, it is often forgotten that first order logic (indeed most logic) works only on a particular type of sentence, the declarative sentence, meaning it cannot be used to infer from question, exclamations, or imperative statements, all of which make frequent appearances in people's daily lives.  The theory that all sentances can be put in declarative form has never been satisfactorily demonstrated.

Equally important is to realize that the assertion of a logical system, as such, is that the inferences hold.  In formal logic this means that true statements entered into the argument as principles will never be contradicted, unless by another input statement, in which case the contradiction will be discoverable. The general notion of a logic as a system of inference in general does not yield logic that always has that property.  For most logic in actual use in human society, quite the opposite seems true, an perhaps the rules of inference are actually intended to hide certain types of troublesome contradictions.

No matter how well constructed some rules of inference may or may not be, there is still a fundamental difference between the conclusions logical argument from that system infer, and the claim they are so inferred. For example, the statement, "if life holds an ultimate purpose, then we ought to live by it" is not inferred by first order formal logic. Whatever one suspect of someone holding that inference as true, it does not, in an analysis of logic abstractly, mean the person believes we ought to live by lives ultimate purpose or that there is such a purpose.

Formal logic, especially traditional first order logic, attempts to make only undeniable analytic assertions about inference, based on AND, OR, and NOT.  In reality these are assertions, if widely palatable ones, and in other more natural and more widely used systems of inference the inferences themselves make statements... such as racist rules of inference which make predictions based on race.

One that I would consider a true logician by my own standard will not apply just one allegedly perfect logical system to a given set of facts and statement. The true logician is a collector of logical tools, and if at all competent will have encountered more than just one set of tools.  This logician will have access to many systems of logic, and will analyze any particular collection of  facts with as many systems of logic as can be applied based on context.  The "true logician" thereby also implicitly employs a still more abstract logic, a meta-logic, a characterization on inference itself, amd ways of combining distinct systems of logic, of averaging their results, of using one to judge the other in practical applications as well as academic ones.

A Well Regulated Militia

The neglected part of the second amendment is in bold below:

A well regulated militia being necessary to the security of a free state, the right of the people to keep and bear arms shall not be infringed.

The current view of the supreme court has ruled that the part in bold is not a requirement that means you must be part of a militia. That is the dissent argues that the second amendment is a collective right, for the "well regulated militia" to bear, but the majority has rejected that. I can accept that reading readily enough.

What I can't help but note, however, is that we have no well regulated militia.  We have the right to bear arms, it's one of the most popular rights in America, so where's our well regulated militia?  The dream of a militia is probably long forgotten and even hidden behind what sort of groups call themselves "militias" at the moment.  The original notion for needing a well regulated militia is rooted in a bias against having a standing army.

In one debate at the Constitutional Convention, if I recall correctly, a standing army was likened to an erect penis, it might make things exciting at home but also invites foreign adventures. Strategically, if the notion of a well regulated militia is even possible to execute, it is a very sound and robust defense model.  The fact is it is hard to occupy nations with well armed citizenry, perhaps impossible depending on the criteria one uses to classify an occupation as "successful".

I think it could work, and it could work against seditious groups that believe the second amendment gives them, or was given them to enable, taking up arms against the Republic.  I would far prefer a still more lenient restriction on guns if a gun permit came with militia training, including a way to be called to arms to defend the country in case of invasion, and in order to reduce the need for a standing military.  It's a military system on the model of volunteer fireman.  People very well trained (regulated) and of confirmed loyalty to the Constitution could as far as I'm concerned have armed tanks and armored vehicles, and be relied on for national defense.

Such forces could be deployed much like the National Guard, in times of crisis, whatever the foe to the nation.

The Second Amendment

Unlike what Ben Shapiro said on Piers Morgan recently, the second amendment is not there to enable us to rise up against tyranny   It's true many founding father felt an armed public is good for this, the document in question, the US Constitution, does not at all specify this.  Indeed, it defines such an action as TREASON, and does not even bother to say, "unless the government has become tyrannical".  The US Government, and every employee, has to swear an oath to the Constitution, and the level of tyranny possible is limited only by the Constitution.  If it is impossible to have a tyranny without violating the constitution  then those oaths hold even in tyranny.

Here is the second amendment:

A well regulated militia being necessary to the security of a free state, the right of the people to keep and bear arms shall not be infringed.

In that sentence we see the phrase "A well regulated militia being necessary to the security of a free state".  That phrase operates as a motivating context.  That implies there can be regulation with respect to the goal of this right being a "well regulated militia".  We have no such militia.  If we did, one could imagine you could own an assault rifle, and the government might even buy for you, if you were well trained as part of a well regulated, non-seditious, militia.  Without that, it's still clear that there is regulation mentioned in the very amendment   It is not the advocacy of something for it's own sake, as if guns were as directly good as say "liberty".

Keep in mind I support the second ammendment, I think American ought to be able to keep tanks and rocket launchers.  I live in a Republic which uses a Democratic device, and I'm in the minority on that.  I can deal with that, and perhaps America knows it's not mature enough to be trusted with such implements of destruction... but perhaps if they were part of a well regulated militia it would be better.

Doctrine

The defeat of objectivist and absolute doctrines does not leave us with nothing. To think that way is to give far too much credit to what has been undermined. There is at least one view in the world, yours.

In a sense there is nothing absolute, but there are indicative definitions, that is, "pointing and naming".  In the most general case we point at all our perceptions, and we name them "perceptions".  These include not only so called sense perceptions, but any perception of mood, of a bodily state, of an urge or even the perception of a memory, allway played out in time.  All reason starts with the bodily action, not merely metaphorically, indicating something and naming it.  The rational for the classifications that are being made comes later, perhaps as a verbal description of the alleged boundary line.

Since I see it as information processing, I prefer the verbal description be of the criteria, a set of rules for placing objects in a set, as if sequentially, one candidate, or group of candidates, at a time.  Everything follows from this, and every abstract (intensional or connotative) meaning is drawn from statements about patterns of relationship.  At the physical bottom, we have measurement of mass, extent, or time done always in comparison with a robust object, so the comparison is consistent at least on one side.

ROUGH: Infrastructure Is the Issue

This essay pertains to the current political situation regarding relief funding to New York and New Jersey and the general issue of common ground infrastructure.  Politically, at the instant, this is a particular instance of "spending" which is supported by some republicans but not by others. Those the support it take it as a given, as the sort of exception to "no spending" that most, but not all republicans, support. Upon inspection we find come modern conservatives take themselves to have adopted a position which is universally "against spending". They have generalized their opposition to some joint spending into a principle, which technically includes opposition to things they actually support.

Personally I hold the view that construction and maintenance of common infrastructure is the only necessary office of what we call government. To explain my thinking process as I experience it I will now ascend to a level of rarified abstraction where I can find principles that can be used as construction cranes to move the heavy ideas on the surface below.

A study of philosophy gives one an idea of what sort of things have proved controversial over the ages, which have been debated for thousands of years, and which are therefore not the prime territory in which to find political agreements. They are rather where one ought expect to find long term controversy analogous to the ancient philosophical disputes to which they relate.  To be sure these ancient gems should be talked about, they must enter into political philosophy, for while there is no consensus from history yielding a single answer or attitude, there is in often a well known (if incomplete) general spectrum of attitudes about the given subject to which one ought to make reference. Through study of these ancient controversies we should construct and then study models of these spectra as maps of the public mind. Whenever possible we should admit credit to those positions that hail from parts of the spectrum most foreign to us.

In the political spectrum one encounters a most complex system of interoperation and conflict. The conflict may be potential, actual, or de facto. It is worth mentioning a civil society will desire to remove de facto conflict when the conflict is not truly necessary, but simply incidental, psychological, or historical, such as racial conflict.  For an individual there may be what turns out to be the most difficult of positions to admit validity too, and this is what must be learned.  We must all learn how to grant validity to some -- most -- positions with which we nevertheless differ. The validity of a position must be understood separate from "positions with which I agree".

For me, the most difficult has been to admit that there are well populated regions in the spectrum of public ideas in which the ways of thinking themselves seem to me to involve illogic.  These are cases where the same set of principles do not lead, for two different parties, to the same conclusions.

Where might we find, in such a spectrum, the common ground, where views overlap, and are held in common. In my vanity I think I have found such a mode and rather than be a subtle and unfamiliar thing, newly created, it is instead a common sense notion, so acceptable we survive even while ignoring our implicit agreement for various incidental reasons, against our common best interest.  We ignore it like one can ignore the air. This is the question, "what shall be build together and hold in common?".

Many now operate on a principle of non-cooperation which states in effect that one not cooperate with others on shared common ground except with those for who the entire ground is common, that is, unless one agrees fully and over all with another, one ought not co-operate, for this helps them as well as you, which does not help to make your personal philosophy the common one.

I interpret such a principle of non-cooperation as dogmatism pure and simple. In political philosophy I complain about little other than dogmatism. When asked what I would do as emperor of the world, I answer that I would install a democratic republic... if asked the details of its laws, I have to accept they do not understand democracy as a goal... how it works, that the point is that the people answer such questions for themselves by a process which can find and reduce to common ground their implicit and explicit agreements.

I accept that dogmatism exists in the public spectrum but I also identify it as the primary source of our systematic problems. I struggle in real time against the view that one should not cooperate even where one has agreement in order to protect one's sect.  In contrast it is my view one ought to cooperate where things are held in common, while trying as best one can to allow for separation, to protect the right of individual decision-making in all areas not held in common. Dogmatists stumble over the idea of accepting that there will, and should always be ideas not held in common.  From my lips "libertarianism" is the notion that indeed, there are many such ideas and they can, and are best to, be decided by the individual free person.

Using this perspective I believe we can reduce conflict to those of material consequence. I celebrate any difference that does not interfere with my freedom to be different. That is my version of individuality, and one ought be able to see it is not individuality as non-cooperation.

To move on to a related issue, it also seems clear to me that what we call government ought really pertain to a shared infrastructure.  Our disagreements should subsequently be, primarily  over what we can agree is really needed in a common infrastructure and the criteria by which we agree to judge when we are achieving it. Secondarily we should endeavor for what is widely desired and agree on means to measure its sustainability.

To me positive politics is all about the question, "what infrastructure are we going to build together?". A weakness in some modern conservatives is a belief that they are in general "against spending", implying there is nothing we want to build together since all joint action requires "spending". It is similarly a weakness in some modern liberals that they are in general "against development", but all "social justice" and liberal ideas require infrastructure which must be developed.  They may mistake their full goals, and in the same way most conservatives are not in fact against all joint spending. Even those conservatives most ardently claiming such a general opposition have generally merely taken for granted that the things they believe in funding jointly can be taken for granted as necessities, and excepted from the universal-sounding rule.

For example, it is relatively easy to get most conservatives to admit they support spending on military.  Some will take a tack that they despise military spending on account of the waste, attempting to hold to their anti-spending rubric, but in such a case they are critizing waste, something that can be measured, and doing so on the basis that there is in principle a type of spending on military that is not wasteful.

A further example is that, at the moment, many conservatives advocate putting armed security in schools and that requires spending.  They arrive at this conclusion without even thinking of spending as such. Even minarchists and market anarchists can be pressed to contradict themselves on the issue as they support things such as gold standard currencies or other social policies which require material outlay to realize.

The question is not "should we spend together on common things", which most already accept, the question is "what do we spend it on?" and "by what criteria can we judge our spending really is spent in service to the agreed upon goal?".  This is the first question not in an abstract model of our political system but in a practical model,  where time and chronological order of events plays a key role.  One might analytically start by asking on what grounds a person can be compelled to spend money on common goals, and even if that can ever be justified, and we can ask foundationally by what grounds a person might be attributed membership in a group or if they have to self-recognize such memberships.  But in practicality, we already have a system of taxation and we already spend the money, so the first problem is what to spend the money on, and then arrises the question on how to pool the money in the first place.  

In general I advocate a tiered flat-fee-for-service revenue model (rather than toll-service or income tax model).  Note, however, as is often the case in modeling, an income tax can be modeled as an annual flat-fee-for-service.  The tax remains much the same, but there are some implications from thinking of the tax this way.  One is that taxes would be separated and in principle, one could opt out.  One's "Military Tax" becomes a fee that goes the the military. The principle that one can opt out is not a universal practicality, however, since for example one cannot opt-out of being protected by the US Military from foriegn invasion if one is in the territory of the US.

It seems current conservatives have been argued into a sort of corner where it is difficult for them to speak specifically about which spending they abhor in particular, and they seek to oppose it all, and hope that the spending they do support is so universally supported they don't have to recognize it as spending at all. This is a fool's approach, and they are already producing people that have bought the generality.  Now, from within their own ranks grows opposition to spending they'd like to pretend is not spending.  That conservatives must talk about "entitlements", meaning AFDC but not Public Education, Social Security, and Medicare, and must talk about "wasteful spending" meaning some of the following: Foriegn Aid, the NEA, Corrupt Public Works Projects, but not some of the subsequent: police departments, fire departments, public roads, or natural disaster relief, merely obfuscates the details of which is it that they really support or oppose.

I think most parties in the US support:

1. military spending of some amount
2. road and bridge funding
3. natural disaster relief
4. education
5. government standards for health, safety, commodification 
6. government involvement in police, fire, and medical emergency services
7. the printing or production of currency

In general, I think it fairly straight forward to argue for a social organizational role in any such case where natural individual ingenuity has not found a sustainable business, and the true debates should be how much is spent, by what criteria it is spend, and through what type of revenue system.  

I can fathom extreme positions on these sub issues, such as support only for a domesticly defensive military, or for public education only in the form of standard testing, or disaster relief only on a fee based public insurance revenu scheme, or private contracting of some emergency services, far more easilly than the easily said positions for the abolition of the DOD, the DEA, etc..

I will, in all cases, argue from a proposition that material progress, that worth working for together, can only be a subset of that which has measurable results, even if only by satisfaction polling. To reduce infant mortality, to extend life expentency, to reduce death from war, to reduce crime, to reduce costs, especially in infrastructure key to survival, such as transportation, commodity production, and public health is widely supported common ground.

Our "government" ought to endeavor to improve any such vital need deemed publicly insufficient in those cases where no wise merchant has yet devised a personally profitable way to improve it naturally with human ingenuity in a self-sustaining way.  That is not to say in such a case that there is no self sustaining way for the need to be provided for profitably and by private means, but merely that when a common need is presented, if there happens to be no such entrant at the time, that the government, so  called, must construct a system as self sustaining as possible. This government ought to also attempt to free the market, give entrance to innovators.

As an example, a postal service of some sort is a necessity in a society and when it cannot be provided at a profit by private parties, the government has stepped in from time immemorial.  Though this was true for centuries, it's clear that now in a modern urban environment, such communication can and should have private providers as well.  Participation in infrastructure should enable such private interoperation as soon as possible, even if that is centuries in coming.