DISCRETE MARKOV DECISION PROBLEM

IN SERVICE FACILITY SYSTEMS WITH INVENTORY

22 th January, 2019.

Markov decision model is a useful and powerful tool for understanding probabilistic sequential decision processes with an infinite planning horizon. Focusing on a discrete-time MDP model of Admission and Inventory Control in Service Facility Systems, the time of operation of the system is divided into periods of time t >0. Decisions are taken at the beginning of each periods (epochs) to control both admission to service and inventory replenishment. Assume that we have 2 kinds of queue, including eligible queue and potential queue. Customers is transferred by the Admission control system at decision epochs from potential queue to eligible queue, either reject or admit. The demand for the services and the service times are assumed to have time invariant probability distributions g() (the arrival of customer in each periods) and f() respectively. Let the revenue be constant throughout the time period. There are 3 types of cost. The system operator earns R for every completely served customer and there is a charge when holding x items in inventory and when there are y customers in the system, i.e. h(x) and k(y) respectively. The maximum inventory is assumed to be M. The MDP model takes account on average cost to find the optimal policy to be implemented for the system.
Let denote be the number of customer in the system before decision epoch t, including number of people in eligible queue and server, and is the number of customer in potential queue in period t. Let be the number of “possible service completions” depending only on the service distribution (not take consideration into the number of customers in the system). We consider the problem on MDP having five components (tuples) .
Decision epochs:

;

State space:
.

where is the number of customer in the system, and is the number of items in stock .
Action: The (M-1, M) policy is adopted for the inventory system. Replenishing is instantaneous and delivery occurs instantaneously when making a decision to order more stock at the beginning of each period.
where 0- no order/reject; 1- order/accept (how many) and 2 –compulsory order/ compulsory accept, depending on the number of \(s_1\) and \(s_2\).
Cost:
The expected number of service completion in period t is
.

Transition probability: is the probability moving from \(s = (s_1, s_2)\) to \(s'= (s_1', s_2')\) under the action \(a \in \mathbb{A}\).
Let denote the state of the system of decision epoch t. Then is a Markov chain with discrete state space S.
The average cost function is alwsays exits (see Theorem (Puterman(1994) & Tijims(2003))), where the denote the total expected cost over the first t decision with the initial state s and policy R is adopted.
The objective is to find the average cost optimal policy which is for each stationary policy R. To see more detail about the algorithm, please see in the reference.
Let’s see one example for illustration. For the system we are N = 5, M = 5. Let the state space be . Assume the holding cost for admission control at level are . Assume that the ordering cost of inventory machine at level are , the holding cost (per inventory), and inventory cost = 0.3(per item). Based on the algorithm, the optimal policies (Admission control) and (inventory control). This mean that at state (0,0) compulsory admission and replenishment, only at the states 1,2,3,4 allow customers to the system and replenishment order is placed when the system state is in state 1 (inventory level).
Reference:
1. Discrete MDP problem with Admission and Inventory Control in Service Facility Systems by C. Selvakumar, P. Maheswari, and C. Elango Research Department of Mathematics, Cardamom Planters’ Association College, Bodinayakanur- 625 513.
2. Optimal Service Control in a Discrete Time Service Facility System with Inventory, Selvakumar, C. 1, Elango, C.2, International Journal of Engineering Science Invention (IJESI) ISSN (Online): 2319 – 6734, ISSN (Print): 2319 – 6726 www.ijesi.org ||Volume 7 Issue 6 Ver I || June 2018 || PP 21-34.
3. Inventory Ordering Control for a Retrial Service Facility System – Semi- MDP, S.Krishnakumar 1, C.Selvakumar3, C.Elango2, International Journal of Engineering Science Invention (IJESI) ISSN (Online): 2319 – 6734, ISSN (Print): 2319 – 6726 www.ijesi.org ||Volume 7 Issue 6 Ver I || June 2018 || PP 14-20.



Comments Please: