~hc/RK356X_SDK_RELEASE.git

..	..	@@ -198,6 +198,15 @@
198	198	prev_comm ~ "sh"
199	199	prev_comm ~ "ba*sh"
200	200
	201	+If the field is a pointer that points into user space (for example
	202	+"filename" from sys_enter_openat), then you have to append ".ustring" to the
	203	+field name::
	204	+
	205	+ filename.ustring ~ "password"
	206	+
	207	+As the kernel will have to know how to retrieve the memory that the pointer
	208	+is at from user space.
	209	+
201	210	5.2 Setting filters
202	211	-------------------
203	212
..	..	@@ -229,6 +238,16 @@
229	238	Currently the caret ('^') for an error always appears at the beginning of
230	239	the filter string; the error message should still be useful though
231	240	even without more accurate position info.
	241	+
	242	+5.2.1 Filter limitations
	243	+------------------------
	244	+
	245	+If a filter is placed on a string pointer ``(char *)`` that does not point
	246	+to a string on the ring buffer, but instead points to kernel or user space
	247	+memory, then, for safety reasons, at most 1024 bytes of the content is
	248	+copied onto a temporary buffer to do the compare. If the copy of the memory
	249	+faults (the pointer points to memory that should not be accessed), then the
	250	+string compare will be treated as not matching.
232	251
233	252	5.3 Clearing filters
234	253	--------------------
..	..	@@ -342,7 +361,8 @@
342	361	differences and the implementation isn't currently tied to it in any
343	362	way, so beware about making generalizations between the two.
344	363
345		-Note: Writing into trace_marker (See Documentation/trace/ftrace.rst)
	364	+.. Note::
	365	+ Writing into trace_marker (See Documentation/trace/ftrace.rst)
346	366	can also enable triggers that are written into
347	367	/sys/kernel/tracing/events/ftrace/print/trigger
348	368
..	..	@@ -525,3 +545,529 @@
525	545	event counts (hitcount).
526	546
527	547	See Documentation/trace/histogram.rst for details and examples.
	548	+
	549	+7. In-kernel trace event API
	550	+============================
	551	+
	552	+In most cases, the command-line interface to trace events is more than
	553	+sufficient. Sometimes, however, applications might find the need for
	554	+more complex relationships than can be expressed through a simple
	555	+series of linked command-line expressions, or putting together sets of
	556	+commands may be simply too cumbersome. An example might be an
	557	+application that needs to 'listen' to the trace stream in order to
	558	+maintain an in-kernel state machine detecting, for instance, when an
	559	+illegal kernel state occurs in the scheduler.
	560	+
	561	+The trace event subsystem provides an in-kernel API allowing modules
	562	+or other kernel code to generate user-defined 'synthetic' events at
	563	+will, which can be used to either augment the existing trace stream
	564	+and/or signal that a particular important state has occurred.
	565	+
	566	+A similar in-kernel API is also available for creating kprobe and
	567	+kretprobe events.
	568	+
	569	+Both the synthetic event and k/ret/probe event APIs are built on top
	570	+of a lower-level "dynevent_cmd" event command API, which is also
	571	+available for more specialized applications, or as the basis of other
	572	+higher-level trace event APIs.
	573	+
	574	+The API provided for these purposes is describe below and allows the
	575	+following:
	576	+
	577	+ - dynamically creating synthetic event definitions
	578	+ - dynamically creating kprobe and kretprobe event definitions
	579	+ - tracing synthetic events from in-kernel code
	580	+ - the low-level "dynevent_cmd" API
	581	+
	582	+7.1 Dyamically creating synthetic event definitions
	583	+---------------------------------------------------
	584	+
	585	+There are a couple ways to create a new synthetic event from a kernel
	586	+module or other kernel code.
	587	+
	588	+The first creates the event in one step, using synth_event_create().
	589	+In this method, the name of the event to create and an array defining
	590	+the fields is supplied to synth_event_create(). If successful, a
	591	+synthetic event with that name and fields will exist following that
	592	+call. For example, to create a new "schedtest" synthetic event::
	593	+
	594	+ ret = synth_event_create("schedtest", sched_fields,
	595	+ ARRAY_SIZE(sched_fields), THIS_MODULE);
	596	+
	597	+The sched_fields param in this example points to an array of struct
	598	+synth_field_desc, each of which describes an event field by type and
	599	+name::
	600	+
	601	+ static struct synth_field_desc sched_fields[] = {
	602	+ { .type = "pid_t", .name = "next_pid_field" },
	603	+ { .type = "char[16]", .name = "next_comm_field" },
	604	+ { .type = "u64", .name = "ts_ns" },
	605	+ { .type = "u64", .name = "ts_ms" },
	606	+ { .type = "unsigned int", .name = "cpu" },
	607	+ { .type = "char[64]", .name = "my_string_field" },
	608	+ { .type = "int", .name = "my_int_field" },
	609	+ };
	610	+
	611	+See synth_field_size() for available types.
	612	+
	613	+If field_name contains [n], the field is considered to be a static array.
	614	+
	615	+If field_names contains[] (no subscript), the field is considered to
	616	+be a dynamic array, which will only take as much space in the event as
	617	+is required to hold the array.
	618	+
	619	+Because space for an event is reserved before assigning field values
	620	+to the event, using dynamic arrays implies that the piecewise
	621	+in-kernel API described below can't be used with dynamic arrays. The
	622	+other non-piecewise in-kernel APIs can, however, be used with dynamic
	623	+arrays.
	624	+
	625	+If the event is created from within a module, a pointer to the module
	626	+must be passed to synth_event_create(). This will ensure that the
	627	+trace buffer won't contain unreadable events when the module is
	628	+removed.
	629	+
	630	+At this point, the event object is ready to be used for generating new
	631	+events.
	632	+
	633	+In the second method, the event is created in several steps. This
	634	+allows events to be created dynamically and without the need to create
	635	+and populate an array of fields beforehand.
	636	+
	637	+To use this method, an empty or partially empty synthetic event should
	638	+first be created using synth_event_gen_cmd_start() or
	639	+synth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(),
	640	+the name of the event along with one or more pairs of args each pair
	641	+representing a 'type field_name;' field specification should be
	642	+supplied. For synth_event_gen_cmd_array_start(), the name of the
	643	+event along with an array of struct synth_field_desc should be
	644	+supplied. Before calling synth_event_gen_cmd_start() or
	645	+synth_event_gen_cmd_array_start(), the user should create and
	646	+initialize a dynevent_cmd object using synth_event_cmd_init().
	647	+
	648	+For example, to create a new "schedtest" synthetic event with two
	649	+fields::
	650	+
	651	+ struct dynevent_cmd cmd;
	652	+ char *buf;
	653	+
	654	+ /* Create a buffer to hold the generated command */
	655	+ buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
	656	+
	657	+ /* Before generating the command, initialize the cmd object */
	658	+ synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
	659	+
	660	+ ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE,
	661	+ "pid_t", "next_pid_field",
	662	+ "u64", "ts_ns");
	663	+
	664	+Alternatively, using an array of struct synth_field_desc fields
	665	+containing the same information::
	666	+
	667	+ ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE,
	668	+ fields, n_fields);
	669	+
	670	+Once the synthetic event object has been created, it can then be
	671	+populated with more fields. Fields are added one by one using
	672	+synth_event_add_field(), supplying the dynevent_cmd object, a field
	673	+type, and a field name. For example, to add a new int field named
	674	+"intfield", the following call should be made::
	675	+
	676	+ ret = synth_event_add_field(&cmd, "int", "intfield");
	677	+
	678	+See synth_field_size() for available types. If field_name contains [n]
	679	+the field is considered to be an array.
	680	+
	681	+A group of fields can also be added all at once using an array of
	682	+synth_field_desc with add_synth_fields(). For example, this would add
	683	+just the first four sched_fields::
	684	+
	685	+ ret = synth_event_add_fields(&cmd, sched_fields, 4);
	686	+
	687	+If you already have a string of the form 'type field_name',
	688	+synth_event_add_field_str() can be used to add it as-is; it will
	689	+also automatically append a ';' to the string.
	690	+
	691	+Once all the fields have been added, the event should be finalized and
	692	+registered by calling the synth_event_gen_cmd_end() function::
	693	+
	694	+ ret = synth_event_gen_cmd_end(&cmd);
	695	+
	696	+At this point, the event object is ready to be used for tracing new
	697	+events.
	698	+
	699	+7.2 Tracing synthetic events from in-kernel code
	700	+------------------------------------------------
	701	+
	702	+To trace a synthetic event, there are several options. The first
	703	+option is to trace the event in one call, using synth_event_trace()
	704	+with a variable number of values, or synth_event_trace_array() with an
	705	+array of values to be set. A second option can be used to avoid the
	706	+need for a pre-formed array of values or list of arguments, via
	707	+synth_event_trace_start() and synth_event_trace_end() along with
	708	+synth_event_add_next_val() or synth_event_add_val() to add the values
	709	+piecewise.
	710	+
	711	+7.2.1 Tracing a synthetic event all at once
	712	+-------------------------------------------
	713	+
	714	+To trace a synthetic event all at once, the synth_event_trace() or
	715	+synth_event_trace_array() functions can be used.
	716	+
	717	+The synth_event_trace() function is passed the trace_event_file
	718	+representing the synthetic event (which can be retrieved using
	719	+trace_get_event_file() using the synthetic event name, "synthetic" as
	720	+the system name, and the trace instance name (NULL if using the global
	721	+trace array)), along with an variable number of u64 args, one for each
	722	+synthetic event field, and the number of values being passed.
	723	+
	724	+So, to trace an event corresponding to the synthetic event definition
	725	+above, code like the following could be used::
	726	+
	727	+ ret = synth_event_trace(create_synth_test, 7, /* number of values */
	728	+ 444, /* next_pid_field */
	729	+ (u64)"clackers", /* next_comm_field */
	730	+ 1000000, /* ts_ns */
	731	+ 1000, /* ts_ms */
	732	+ smp_processor_id(),/* cpu */
	733	+ (u64)"Thneed", /* my_string_field */
	734	+ 999); /* my_int_field */
	735	+
	736	+All vals should be cast to u64, and string vals are just pointers to
	737	+strings, cast to u64. Strings will be copied into space reserved in
	738	+the event for the string, using these pointers.
	739	+
	740	+Alternatively, the synth_event_trace_array() function can be used to
	741	+accomplish the same thing. It is passed the trace_event_file
	742	+representing the synthetic event (which can be retrieved using
	743	+trace_get_event_file() using the synthetic event name, "synthetic" as
	744	+the system name, and the trace instance name (NULL if using the global
	745	+trace array)), along with an array of u64, one for each synthetic
	746	+event field.
	747	+
	748	+To trace an event corresponding to the synthetic event definition
	749	+above, code like the following could be used::
	750	+
	751	+ u64 vals[7];
	752	+
	753	+ vals[0] = 777; /* next_pid_field */
	754	+ vals[1] = (u64)"tiddlywinks"; /* next_comm_field */
	755	+ vals[2] = 1000000; /* ts_ns */
	756	+ vals[3] = 1000; /* ts_ms */
	757	+ vals[4] = smp_processor_id(); /* cpu */
	758	+ vals[5] = (u64)"thneed"; /* my_string_field */
	759	+ vals[6] = 398; /* my_int_field */
	760	+
	761	+The 'vals' array is just an array of u64, the number of which must
	762	+match the number of field in the synthetic event, and which must be in
	763	+the same order as the synthetic event fields.
	764	+
	765	+All vals should be cast to u64, and string vals are just pointers to
	766	+strings, cast to u64. Strings will be copied into space reserved in
	767	+the event for the string, using these pointers.
	768	+
	769	+In order to trace a synthetic event, a pointer to the trace event file
	770	+is needed. The trace_get_event_file() function can be used to get
	771	+it - it will find the file in the given trace instance (in this case
	772	+NULL since the top trace array is being used) while at the same time
	773	+preventing the instance containing it from going away::
	774	+
	775	+ schedtest_event_file = trace_get_event_file(NULL, "synthetic",
	776	+ "schedtest");
	777	+
	778	+Before tracing the event, it should be enabled in some way, otherwise
	779	+the synthetic event won't actually show up in the trace buffer.
	780	+
	781	+To enable a synthetic event from the kernel, trace_array_set_clr_event()
	782	+can be used (which is not specific to synthetic events, so does need
	783	+the "synthetic" system name to be specified explicitly).
	784	+
	785	+To enable the event, pass 'true' to it::
	786	+
	787	+ trace_array_set_clr_event(schedtest_event_file->tr,
	788	+ "synthetic", "schedtest", true);
	789	+
	790	+To disable it pass false::
	791	+
	792	+ trace_array_set_clr_event(schedtest_event_file->tr,
	793	+ "synthetic", "schedtest", false);
	794	+
	795	+Finally, synth_event_trace_array() can be used to actually trace the
	796	+event, which should be visible in the trace buffer afterwards::
	797	+
	798	+ ret = synth_event_trace_array(schedtest_event_file, vals,
	799	+ ARRAY_SIZE(vals));
	800	+
	801	+To remove the synthetic event, the event should be disabled, and the
	802	+trace instance should be 'put' back using trace_put_event_file()::
	803	+
	804	+ trace_array_set_clr_event(schedtest_event_file->tr,
	805	+ "synthetic", "schedtest", false);
	806	+ trace_put_event_file(schedtest_event_file);
	807	+
	808	+If those have been successful, synth_event_delete() can be called to
	809	+remove the event::
	810	+
	811	+ ret = synth_event_delete("schedtest");
	812	+
	813	+7.2.2 Tracing a synthetic event piecewise
	814	+-----------------------------------------
	815	+
	816	+To trace a synthetic using the piecewise method described above, the
	817	+synth_event_trace_start() function is used to 'open' the synthetic
	818	+event trace::
	819	+
	820	+ struct synth_trace_state trace_state;
	821	+
	822	+ ret = synth_event_trace_start(schedtest_event_file, &trace_state);
	823	+
	824	+It's passed the trace_event_file representing the synthetic event
	825	+using the same methods as described above, along with a pointer to a
	826	+struct synth_trace_state object, which will be zeroed before use and
	827	+used to maintain state between this and following calls.
	828	+
	829	+Once the event has been opened, which means space for it has been
	830	+reserved in the trace buffer, the individual fields can be set. There
	831	+are two ways to do that, either one after another for each field in
	832	+the event, which requires no lookups, or by name, which does. The
	833	+tradeoff is flexibility in doing the assignments vs the cost of a
	834	+lookup per field.
	835	+
	836	+To assign the values one after the other without lookups,
	837	+synth_event_add_next_val() should be used. Each call is passed the
	838	+same synth_trace_state object used in the synth_event_trace_start(),
	839	+along with the value to set the next field in the event. After each
	840	+field is set, the 'cursor' points to the next field, which will be set
	841	+by the subsequent call, continuing until all the fields have been set
	842	+in order. The same sequence of calls as in the above examples using
	843	+this method would be (without error-handling code)::
	844	+
	845	+ /* next_pid_field */
	846	+ ret = synth_event_add_next_val(777, &trace_state);
	847	+
	848	+ /* next_comm_field */
	849	+ ret = synth_event_add_next_val((u64)"slinky", &trace_state);
	850	+
	851	+ /* ts_ns */
	852	+ ret = synth_event_add_next_val(1000000, &trace_state);
	853	+
	854	+ /* ts_ms */
	855	+ ret = synth_event_add_next_val(1000, &trace_state);
	856	+
	857	+ /* cpu */
	858	+ ret = synth_event_add_next_val(smp_processor_id(), &trace_state);
	859	+
	860	+ /* my_string_field */
	861	+ ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state);
	862	+
	863	+ /* my_int_field */
	864	+ ret = synth_event_add_next_val(395, &trace_state);
	865	+
	866	+To assign the values in any order, synth_event_add_val() should be
	867	+used. Each call is passed the same synth_trace_state object used in
	868	+the synth_event_trace_start(), along with the field name of the field
	869	+to set and the value to set it to. The same sequence of calls as in
	870	+the above examples using this method would be (without error-handling
	871	+code)::
	872	+
	873	+ ret = synth_event_add_val("next_pid_field", 777, &trace_state);
	874	+ ret = synth_event_add_val("next_comm_field", (u64)"silly putty",
	875	+ &trace_state);
	876	+ ret = synth_event_add_val("ts_ns", 1000000, &trace_state);
	877	+ ret = synth_event_add_val("ts_ms", 1000, &trace_state);
	878	+ ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state);
	879	+ ret = synth_event_add_val("my_string_field", (u64)"thneed_9",
	880	+ &trace_state);
	881	+ ret = synth_event_add_val("my_int_field", 3999, &trace_state);
	882	+
	883	+Note that synth_event_add_next_val() and synth_event_add_val() are
	884	+incompatible if used within the same trace of an event - either one
	885	+can be used but not both at the same time.
	886	+
	887	+Finally, the event won't be actually traced until it's 'closed',
	888	+which is done using synth_event_trace_end(), which takes only the
	889	+struct synth_trace_state object used in the previous calls::
	890	+
	891	+ ret = synth_event_trace_end(&trace_state);
	892	+
	893	+Note that synth_event_trace_end() must be called at the end regardless
	894	+of whether any of the add calls failed (say due to a bad field name
	895	+being passed in).
	896	+
	897	+7.3 Dyamically creating kprobe and kretprobe event definitions
	898	+--------------------------------------------------------------
	899	+
	900	+To create a kprobe or kretprobe trace event from kernel code, the
	901	+kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start()
	902	+functions can be used.
	903	+
	904	+To create a kprobe event, an empty or partially empty kprobe event
	905	+should first be created using kprobe_event_gen_cmd_start(). The name
	906	+of the event and the probe location should be specfied along with one
	907	+or args each representing a probe field should be supplied to this
	908	+function. Before calling kprobe_event_gen_cmd_start(), the user
	909	+should create and initialize a dynevent_cmd object using
	910	+kprobe_event_cmd_init().
	911	+
	912	+For example, to create a new "schedtest" kprobe event with two fields::
	913	+
	914	+ struct dynevent_cmd cmd;
	915	+ char *buf;
	916	+
	917	+ /* Create a buffer to hold the generated command */
	918	+ buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
	919	+
	920	+ /* Before generating the command, initialize the cmd object */
	921	+ kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
	922	+
	923	+ /*
	924	+ * Define the gen_kprobe_test event with the first 2 kprobe
	925	+ * fields.
	926	+ */
	927	+ ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open",
	928	+ "dfd=%ax", "filename=%dx");
	929	+
	930	+Once the kprobe event object has been created, it can then be
	931	+populated with more fields. Fields can be added using
	932	+kprobe_event_add_fields(), supplying the dynevent_cmd object along
	933	+with a variable arg list of probe fields. For example, to add a
	934	+couple additional fields, the following call could be made::
	935	+
	936	+ ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");
	937	+
	938	+Once all the fields have been added, the event should be finalized and
	939	+registered by calling the kprobe_event_gen_cmd_end() or
	940	+kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe
	941	+or kretprobe command was started::
	942	+
	943	+ ret = kprobe_event_gen_cmd_end(&cmd);
	944	+
	945	+or::
	946	+
	947	+ ret = kretprobe_event_gen_cmd_end(&cmd);
	948	+
	949	+At this point, the event object is ready to be used for tracing new
	950	+events.
	951	+
	952	+Similarly, a kretprobe event can be created using
	953	+kretprobe_event_gen_cmd_start() with a probe name and location and
	954	+additional params such as $retval::
	955	+
	956	+ ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test",
	957	+ "do_sys_open", "$retval");
	958	+
	959	+Similar to the synthetic event case, code like the following can be
	960	+used to enable the newly created kprobe event::
	961	+
	962	+ gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test");
	963	+
	964	+ ret = trace_array_set_clr_event(gen_kprobe_test->tr,
	965	+ "kprobes", "gen_kprobe_test", true);
	966	+
	967	+Finally, also similar to synthetic events, the following code can be
	968	+used to give the kprobe event file back and delete the event::
	969	+
	970	+ trace_put_event_file(gen_kprobe_test);
	971	+
	972	+ ret = kprobe_event_delete("gen_kprobe_test");
	973	+
	974	+7.4 The "dynevent_cmd" low-level API
	975	+------------------------------------
	976	+
	977	+Both the in-kernel synthetic event and kprobe interfaces are built on
	978	+top of a lower-level "dynevent_cmd" interface. This interface is
	979	+meant to provide the basis for higher-level interfaces such as the
	980	+synthetic and kprobe interfaces, which can be used as examples.
	981	+
	982	+The basic idea is simple and amounts to providing a general-purpose
	983	+layer that can be used to generate trace event commands. The
	984	+generated command strings can then be passed to the command-parsing
	985	+and event creation code that already exists in the trace event
	986	+subystem for creating the corresponding trace events.
	987	+
	988	+In a nutshell, the way it works is that the higher-level interface
	989	+code creates a struct dynevent_cmd object, then uses a couple
	990	+functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up
	991	+a command string, which finally causes the command to be executed
	992	+using the dynevent_create() function. The details of the interface
	993	+are described below.
	994	+
	995	+The first step in building a new command string is to create and
	996	+initialize an instance of a dynevent_cmd. Here, for instance, we
	997	+create a dynevent_cmd on the stack and initialize it::
	998	+
	999	+ struct dynevent_cmd cmd;
	1000	+ char *buf;
	1001	+ int ret;
	1002	+
	1003	+ buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
	1004	+
	1005	+ dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO,
	1006	+ foo_event_run_command);
	1007	+
	1008	+The dynevent_cmd initialization needs to be given a user-specified
	1009	+buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used
	1010	+for this purpose - at 2k it's generally too big to be comfortably put
	1011	+on the stack, so is dynamically allocated), a dynevent type id, which
	1012	+is meant to be used to check that further API calls are for the
	1013	+correct command type, and a pointer to an event-specific run_command()
	1014	+callback that will be called to actually execute the event-specific
	1015	+command function.
	1016	+
	1017	+Once that's done, the command string can by built up by successive
	1018	+calls to argument-adding functions.
	1019	+
	1020	+To add a single argument, define and initialize a struct dynevent_arg
	1021	+or struct dynevent_arg_pair object. Here's an example of the simplest
	1022	+possible arg addition, which is simply to append the given string as
	1023	+a whitespace-separated argument to the command::
	1024	+
	1025	+ struct dynevent_arg arg;
	1026	+
	1027	+ dynevent_arg_init(&arg, NULL, 0);
	1028	+
	1029	+ arg.str = name;
	1030	+
	1031	+ ret = dynevent_arg_add(cmd, &arg);
	1032	+
	1033	+The arg object is first initialized using dynevent_arg_init() and in
	1034	+this case the parameters are NULL or 0, which means there's no
	1035	+optional sanity-checking function or separator appended to the end of
	1036	+the arg.
	1037	+
	1038	+Here's another more complicated example using an 'arg pair', which is
	1039	+used to create an argument that consists of a couple components added
	1040	+together as a unit, for example, a 'type field_name;' arg or a simple
	1041	+expression arg e.g. 'flags=%cx'::
	1042	+
	1043	+ struct dynevent_arg_pair arg_pair;
	1044	+
	1045	+ dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';');
	1046	+
	1047	+ arg_pair.lhs = type;
	1048	+ arg_pair.rhs = name;
	1049	+
	1050	+ ret = dynevent_arg_pair_add(cmd, &arg_pair);
	1051	+
	1052	+Again, the arg_pair is first initialized, in this case with a callback
	1053	+function used to check the sanity of the args (for example, that
	1054	+neither part of the pair is NULL), along with a character to be used
	1055	+to add an operator between the pair (here none) and a separator to be
	1056	+appended onto the end of the arg pair (here ';').
	1057	+
	1058	+There's also a dynevent_str_add() function that can be used to simply
	1059	+add a string as-is, with no spaces, delimeters, or arg check.
	1060	+
	1061	+Any number of dynevent_*_add() calls can be made to build up the string
	1062	+(until its length surpasses cmd->maxlen). When all the arguments have
	1063	+been added and the command string is complete, the only thing left to
	1064	+do is run the command, which happens by simply calling
	1065	+dynevent_create()::
	1066	+
	1067	+ ret = dynevent_create(&cmd);
	1068	+
	1069	+At that point, if the return value is 0, the dynamic event has been
	1070	+created and is ready to use.
	1071	+
	1072	+See the dynevent_cmd function definitions themselves for the details
	1073	+of the API.